Oct 30 20:36:31.740212 ip-10-0-128-161 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:36:32.170365 ip-10-0-128-161 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:36:32.170365 ip-10-0-128-161 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:36:32.170365 ip-10-0-128-161 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:36:32.170365 ip-10-0-128-161 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:36:32.175291 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175210    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:36:32.175291 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175291    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:36:32.175291 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175294    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175297    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175301    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175304    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175307    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175309    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175312    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175314    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175317    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175320    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175322    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175324    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175334    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175337    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175340    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175342    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175344    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175347    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175349    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:36:32.175380 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175352    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175354    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175357    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175360    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175362    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175364    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175367    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175369    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175372    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175374    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175377    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175379    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175382    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175386    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175392    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175396    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175399    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175402    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175405    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:36:32.175873 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175407    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175410    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175413    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175415    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175418    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175420    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175423    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175425    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175428    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175430    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175433    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175435    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175438    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175440    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175442    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175446    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175449    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175451    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175454    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175457    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:36:32.176340 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175460    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175462    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175464    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175467    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175470    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175473    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175475    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175478    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175480    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175482    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175485    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175487    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175494    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175497    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175499    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175502    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175504    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175508    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175512    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:36:32.176878 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175515    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175517    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175520    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175522    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175525    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175527    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175530    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175532    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175535    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175538    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175541    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175543    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175546    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175548    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175551    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175554    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175556    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175559    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175561    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175564    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:36:32.177642 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175566    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175569    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175572    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175574    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175576    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175580    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.175583    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178237    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178254    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178260    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178265    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178269    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178274    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178278    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178282    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178286    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178290    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178294    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178299    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178303    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:36:32.178452 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178308    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178312    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178317    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178321    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178325    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178329    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178334    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178338    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178343    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178347    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178351    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178355    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178359    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178364    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178368    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178372    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178376    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178380    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178385    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:36:32.179283 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178389    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178394    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178398    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178402    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178406    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178410    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178414    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178418    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178422    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178427    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178431    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178437    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178444    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178450    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178455    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178459    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178464    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178468    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:36:32.180121 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178472    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178476    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178481    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178487    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178491    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178496    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178500    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178504    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178509    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178513    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178517    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178522    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178526    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178530    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178534    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178539    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178545    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178551    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178556    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:36:32.180582 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178562    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178567    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178571    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178575    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178579    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178583    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178587    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178591    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178595    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178600    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178604    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178608    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178612    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178616    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178622    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178626    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178630    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178634    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178637    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:36:32.181052 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178642    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178646    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178650    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178654    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178658    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178662    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178667    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178670    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178675    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178680    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178685    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178689    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178693    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178697    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178701    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178705    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178709    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.178713    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179527    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179536    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:36:32.181733 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179540    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179544    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179549    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179553    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179557    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179561    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179566    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179570    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179575    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179579    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179583    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179587    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179591    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179596    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179601    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179605    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179609    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179613    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179617    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179621    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:36:32.182587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179625    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179630    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179634    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179638    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179646    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179653    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179659    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179664    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179670    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179675    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179682    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179689    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179694    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179698    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179702    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179707    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179711    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179715    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179720    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:36:32.183433 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179724    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179729    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179733    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179737    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179741    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179745    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179749    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179754    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179758    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179762    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179766    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179770    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179774    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179779    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179783    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179788    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179792    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179795    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179800    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179804    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:36:32.184034 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179808    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179812    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179816    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179820    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179824    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179829    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179833    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179837    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179841    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179845    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179849    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179853    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179874    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179878    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179883    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179887    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179891    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179897    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179902    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179906    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:36:32.184714 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179909    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179914    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179918    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179922    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179926    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179930    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179935    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179939    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179943    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179946    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179951    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179955    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179959    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179963    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179967    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179971    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179975    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179979    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179983    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:36:32.185211 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179987    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179992    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.179996    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.180000    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.180004    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.180010    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181240    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181256    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181270    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181277    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181284    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181289    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181295    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181302    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181320    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181325    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181330    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181336    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181341    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181346    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181351    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181356    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181360    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:36:32.185771 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181365    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181370    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181375    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181380    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181385    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181390    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181396    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181401    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181406    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181411    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181415    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181420    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181425    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181430    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181437    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181445    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181450    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181454    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181460    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181465    2576 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181469    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181477    2576 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181482    2576 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181487    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:36:32.186397 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181492    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181496    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181503    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181507    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181513    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181518    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181523    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181527    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181532    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181536    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181541    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181545    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181550    2576 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181557    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181561    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181567    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181573    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181577    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181582    2576 flags.go:64] FLAG: --help="false"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181587    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181592    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181597    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181603    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:36:32.186999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181608    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181613    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181618    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181623    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181629    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181634    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181639    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181644    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181648    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181652    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181657    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181662    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181667    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181671    2576 flags.go:64] FLAG: --lock-file=""
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181676    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181680    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181685    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181701    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181706    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181711    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181716    2576 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181720    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181726    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181730    2576 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181734    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:36:32.187519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181742    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181746    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181752    2576 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181757    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181762    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181766    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181772    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181777    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181782    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181786    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181801    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181808    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181814    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181818    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181823    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181827    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181832    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181836    2576 flags.go:64] FLAG: --port="10250"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181841    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181846    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0056c41d4d4faca06"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181851    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181870    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181875    2576 flags.go:64] FLAG: --register-node="true"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181880    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181892    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181896    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:36:32.188149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181900    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181905    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181911    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181917    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181921    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181926    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181930    2576 flags.go:64] FLAG: --runonce="false"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181935    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181940    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181945    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181949    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181954    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181959    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181964    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181969    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181973    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181978    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181982    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181987    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181992    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.181997    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182002    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182010    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182015    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182019    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:36:32.188796 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182025    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182029    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182034    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182038    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182043    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182048    2576 flags.go:64] FLAG: --v="2"
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182055    2576 flags.go:64] FLAG: --version="false"
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182061    2576 flags.go:64] FLAG: --vmodule=""
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182067    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182072    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182303    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182311    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182317    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182322    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182326    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182332    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182336    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182341    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182344    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182352    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182358    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:36:32.189453 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182363    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182367    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182372    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182377    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182381    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182384    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182390    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182395    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182399    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182403    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182407    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182411    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182416    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182420    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182424    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182430    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182435    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182440    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182444    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182448    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:36:32.189955 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182452    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182457    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182461    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182465    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182470    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182474    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182483    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182487    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182492    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182496    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182500    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182504    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182508    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182512    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182517    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182521    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182525    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182529    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182533    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:36:32.190401 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182539    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182543    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182547    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182552    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182556    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182560    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182564    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182568    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182572    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182576    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182580    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182584    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182588    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182592    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182596    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182600    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182604    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182608    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182612    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:36:32.190844 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182617    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182621    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182626    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182630    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182635    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182639    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182644    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182648    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182652    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182656    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182661    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182665    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182669    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182673    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182678    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182682    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182686    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182690    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182694    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182699    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:36:32.191301 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182703    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182708    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182712    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182716    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182720    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182724    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182728    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182732    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182736    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182740    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182744    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182749    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182754    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182758    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182763    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182767    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:36:32.191752 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182774    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.182812    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182883    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182892    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182897    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182902    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182906    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182910    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182914    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182919    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182923    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182927    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182932    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182936    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182941    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182945    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:36:32.192223 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182948    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182952    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182956    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182960    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182965    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182968    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182973    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182977    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182981    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182985    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182990    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.182994    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183000    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183004    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183008    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183012    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183016    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183021    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183026    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:36:32.192602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183030    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183034    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183038    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183042    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183046    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183051    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183055    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183059    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183063    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183068    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183072    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183076    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183080    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183084    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183089    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183093    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183097    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183102    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183106    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:36:32.193077 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183110    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183114    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183118    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183122    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183126    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183130    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183136    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183142    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183146    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183150    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183154    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183159    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183163    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183167    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183171    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183176    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183180    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183184    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183188    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183192    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:36:32.193522 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183197    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183201    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183205    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183210    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183214    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183219    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183223    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183227    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183231    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183235    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183239    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183243    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183247    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183251    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183258    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183263    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183268    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183272    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183278    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:36:32.193995 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183282    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183286    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183290    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183295    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183301    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183305    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183309    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183313    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183318    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183322    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183326    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183330    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183338    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183344    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.183349    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.190199    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.190214    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.190230    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.190235    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190291    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:36:32.194451 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190296    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190299    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190302    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190305    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190308    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190310    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190313    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190315    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190318    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190320    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190323    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190326    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190328    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190331    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190334    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190336    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190339    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190341    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190344    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190346    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:36:32.194920 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190349    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190351    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190354    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190356    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190358    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190361    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190363    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190366    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190370    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190373    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190376    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190378    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190381    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190383    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190386    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190388    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190391    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190394    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190396    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190399    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:36:32.195369 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190401    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190404    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190407    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190409    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190412    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190415    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190418    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190420    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190423    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190425    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190428    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190431    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190433    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190436    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190439    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190441    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190444    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190447    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190449    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190452    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:36:32.195823 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190455    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190458    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190461    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190463    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190466    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190468    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190471    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190474    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190476    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190479    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190481    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190485    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190489    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190492    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190495    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190497    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190500    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190502    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190505    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:36:32.196342 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190507    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190510    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190512    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190515    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190517    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190520    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190522    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190525    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190528    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190530    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190533    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190536    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190538    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190541    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190543    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190546    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190549    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190551    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190554    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:36:32.196765 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190556    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190559    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190561    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190564    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190566    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190569    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190573    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.190599    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190770    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190774    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190777    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190780    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190783    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190785    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190788    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:36:32.197215 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190792    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190795    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190799    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190801    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190804    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190806    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190809    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190811    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190814    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190816    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190819    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190821    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190823    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190826    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190834    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190837    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190839    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190842    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190844    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:36:32.197601 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190847    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190849    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190852    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190868    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190873    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190877    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190881    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190884    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190888    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190891    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190893    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190896    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190898    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190901    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190903    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190906    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190908    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190910    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190913    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:36:32.198037 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190915    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190918    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190920    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190922    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190925    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190927    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190930    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190932    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190934    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190943    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190945    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190948    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190950    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190953    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190955    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190958    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190960    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190963    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190965    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:36:32.198564 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190967    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190976    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190979    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190981    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190984    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190986    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190988    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190991    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190993    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190996    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.190998    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191001    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191003    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191006    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191008    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191011    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191013    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191015    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191018    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191020    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:36:32.199143 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191022    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191025    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191027    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191035    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191038    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191040    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191043    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191047    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191050    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191053    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191055    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191058    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191060    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191062    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191065    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191067    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191070    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191072    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191074    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191077    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:36:32.199587 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191079    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191082    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.191105    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191146    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191150    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191153    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191156    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191158    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191161    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191164    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191167    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191170    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191173    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191175    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191178    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191180    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:36:32.200233 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191188    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191191    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191193    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191196    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191199    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191201    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191203    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191206    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191208    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191210    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191213    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191215    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191218    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191220    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191223    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191225    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191229    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191233    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191236    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:36:32.200602 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191239    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191242    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191245    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191247    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191249    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191252    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191257    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191259    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191261    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191264    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191266    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191269    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191272    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191274    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191282    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191285    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191287    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191291    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191294    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:36:32.201071 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191296    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191299    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191302    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191304    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191307    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191309    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191312    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191315    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191317    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191320    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191322    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191325    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191327    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191330    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191332    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191334    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191337    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191340    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191342    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:36:32.201620 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191346    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191349    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191351    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191354    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191356    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191359    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191361    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191364    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191366    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191374    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191377    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191381    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191383    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191386    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191388    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191390    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191393    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191396    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191398    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191401    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:36:32.202334 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191403    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191406    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191408    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191410    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191413    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191416    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191418    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191420    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191424    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191426    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191429    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191431    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191435    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191438    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191440    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:32.191443    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.191495    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.193477    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.194601    2576 server.go:1014] "Starting client certificate rotation"
Oct 30 20:36:32.202949 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.194705    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:36:32.203432 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.194744    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:36:32.203432 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.196900    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:36:32.203432 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.198296    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:36:32.210680 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.210665    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:36:32.213976 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.213958    2576 log.go:25] "Validated CRI v1 image API"
Oct 30 20:36:32.214785 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.214762    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:36:32.217383 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.217367    2576 fs.go:135] Filesystem UUIDs: map[02c9c38f-07fd-4d4b-8c39-5600dc070a2f:/dev/nvme0n1p4 76d67411-3c77-4ee0-8ae4-6cbebfe6a044:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 20:36:32.217445 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.217383    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 20:36:32.222437 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.222328    2576 manager.go:217] Machine: {Timestamp:2026-10-30 20:36:32.220810459 +0000 UTC m=+0.356333605 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3181024 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e67722f9f68cd563b728fc8088185 SystemUUID:ec2e6772-2f9f-68cd-563b-728fc8088185 BootID:8e42368f-59cc-4a19-9a7b-bf69fc9f7e6d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:c7:ec:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:c7:ec:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:d6:2a:72:bb:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:36:32.222909 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.222897    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:36:32.223018 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.223006    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:36:32.226130 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.226105    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:36:32.226253 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.226133    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-161.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:36:32.226814 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.226801    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:36:32.226814 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.226812    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:36:32.226911 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.226840    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:36:32.226911 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.226853    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:36:32.227655 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.227645    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:36:32.229106 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.229084    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:36:32.231617 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.231606    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:36:32.231739 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.231729    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:36:32.231783 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.231759    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:36:32.231783 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.231776    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:36:32.231845 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.231786    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:36:32.231845 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.231795    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:36:32.231845 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.231802    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:36:32.232213 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.232200    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:36:32.234866 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.234829    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:36:32.239766 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.239751    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:36:32.240500 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.240487    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:36:32.240566 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.240517    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:36:32.240662 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.240653    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:36:32.240693 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.240668    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:36:32.240693 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.240674    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:36:32.240693 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.240680    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:36:32.240693 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.240686    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:36:32.240693 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.240692    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:36:32.241393 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.241384    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:36:32.241393 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.241393    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:36:32.241447 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.241401    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:36:32.241447 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.241407    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:36:32.241447 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.241415    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:36:32.241447 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.241425    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:36:32.241970 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.241961    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:36:32.242001 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.241971    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:36:32.245654 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.245642    2576 server.go:1263] "Started kubelet"
Oct 30 20:36:32.246772 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.246722    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:36:32.246868 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.246790    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:36:32.247307 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.247279    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:36:32.247550 ip-10-0-128-161 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:36:32.248651 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.247838    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:36:32.250074 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.249557    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-161.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 20:36:32.251945 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.251929    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:36:32.254127 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.254108    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:36:32.254899 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.254878    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 20:36:32.255170 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.255156    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:36:32.255260 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.255238    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 20:36:32.255365 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.255251    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:32.255365 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.255244    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-161.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 20:36:32.255811 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.255794    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:36:32.255811 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.255796    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:36:32.255947 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.255821    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:36:32.255947 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.255891    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:36:32.255947 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.255898    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:36:32.255947 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.255927    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:36:32.255947 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.255937    2576 factory.go:55] Registering systemd factory
Oct 30 20:36:32.255947 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.255943    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:36:32.256233 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.256147    2576 factory.go:153] Registering CRI-O factory
Oct 30 20:36:32.256233 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.256161    2576 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:36:32.256233 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.256175    2576 factory.go:103] Registering Raw factory
Oct 30 20:36:32.256233 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.256187    2576 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:36:32.256499 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.255537    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.us-east-2.compute.internal.18e3690f939df082  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:36:32.24561677 +0000 UTC m=+0.381139914,LastTimestamp:2026-10-30 20:36:32.24561677 +0000 UTC m=+0.381139914,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.us-east-2.compute.internal,}"
Oct 30 20:36:32.257084 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.257068    2576 manager.go:319] Starting recovery of all containers
Oct 30 20:36:32.263635 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.263595    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-161.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 20:36:32.266269 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.266249    2576 manager.go:324] Recovery completed
Oct 30 20:36:32.267700 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.267672    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 20:36:32.271638 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.271616    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:36:32.274049 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.274030    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:36:32.274123 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.274070    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:36:32.274123 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.274093    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:36:32.275044 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.275028    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:36:32.275044 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.275043    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:36:32.275143 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.275058    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:36:32.275143 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.275035    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:36:32.275816 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.275799    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:36:32.275910 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.275831    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:36:32.275910 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.275847    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:36:32.276874 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.276779    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.us-east-2.compute.internal.18e3690f954fcb66  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-161.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:36:32.274049894 +0000 UTC m=+0.409573046,LastTimestamp:2026-10-30 20:36:32.274049894 +0000 UTC m=+0.409573046,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.us-east-2.compute.internal,}"
Oct 30 20:36:32.277332 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.277304    2576 policy_none.go:50] "Start"
Oct 30 20:36:32.277332 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.277333    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:36:32.277452 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.277346    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:36:32.279440 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.279427    2576 policy_none.go:44] "Start"
Oct 30 20:36:32.286635 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.286617    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n6blx"
Oct 30 20:36:32.288123 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.288020    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.us-east-2.compute.internal.18e3690f95504ff3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-161.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:36:32.274083827 +0000 UTC m=+0.409606977,LastTimestamp:2026-10-30 20:36:32.274083827 +0000 UTC m=+0.409606977,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.us-east-2.compute.internal,}"
Oct 30 20:36:32.295514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.295496    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n6blx"
Oct 30 20:36:32.298997 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.298931    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.us-east-2.compute.internal.18e3690f95509056  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-161.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:36:32.27410031 +0000 UTC m=+0.409623459,LastTimestamp:2026-10-30 20:36:32.27410031 +0000 UTC m=+0.409623459,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.us-east-2.compute.internal,}"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.313490    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-161.us-east-2.compute.internal.18e3690f954fcb66\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.us-east-2.compute.internal.18e3690f954fcb66  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-161.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:36:32.274049894 +0000 UTC m=+0.409573046,LastTimestamp:2026-10-30 20:36:32.275815911 +0000 UTC m=+0.411339063,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.us-east-2.compute.internal,}"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.319536    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.319806    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.320934    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.320950    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.320966    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.320994    2576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.324784    2576 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.325106    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-161.us-east-2.compute.internal.18e3690f95504ff3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.us-east-2.compute.internal.18e3690f95504ff3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-161.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:36:32.274083827 +0000 UTC m=+0.409606977,LastTimestamp:2026-10-30 20:36:32.275839053 +0000 UTC m=+0.411362205,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.us-east-2.compute.internal,}"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.325232    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.325246    2576 server.go:87] "Starting device plugin registration server"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.325480    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.325493    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.325559    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.325630    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.325637    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.326542    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.326586    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.332217    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 20:36:32.414058 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.332273    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-161.us-east-2.compute.internal.18e3690f95509056\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.us-east-2.compute.internal.18e3690f95509056  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-161.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:36:32.27410031 +0000 UTC m=+0.409623459,LastTimestamp:2026-10-30 20:36:32.275852278 +0000 UTC m=+0.411375423,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.us-east-2.compute.internal,}"
Oct 30 20:36:32.414659 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.336012    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.us-east-2.compute.internal.18e3690f982cf690  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:36:32.322098832 +0000 UTC m=+0.457621968,LastTimestamp:2026-10-30 20:36:32.322098832 +0000 UTC m=+0.457621968,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.us-east-2.compute.internal,}"
Oct 30 20:36:32.414659 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.341449    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.us-east-2.compute.internal.18e3690f9855863f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:36:32.324757055 +0000 UTC m=+0.460280202,LastTimestamp:2026-10-30 20:36:32.324757055 +0000 UTC m=+0.460280202,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.us-east-2.compute.internal,}"
Oct 30 20:36:32.421266 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.421220    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"]
Oct 30 20:36:32.421373 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.421273    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:36:32.422059 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.422045    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:36:32.422142 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.422072    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:36:32.422142 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.422081    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:36:32.424268 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.424254    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:36:32.424360 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.424251    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-161.us-east-2.compute.internal.18e3690f954fcb66\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-161.us-east-2.compute.internal.18e3690f954fcb66  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-161.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-161.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-161.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:36:32.274049894 +0000 UTC m=+0.409573046,LastTimestamp:2026-10-30 20:36:32.422059276 +0000 UTC m=+0.557582420,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-161.us-east-2.compute.internal,}"
Oct 30 20:36:32.424486 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.424470    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.424551 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.424500    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:36:32.424970 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.424954    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:36:32.425069 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.424980    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:36:32.425069 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.424995    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:36:32.425207 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.425181    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:36:32.425291 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.425219    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:36:32.425291 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.425230    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:36:32.425834 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.425814    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:36:32.426457 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.426442    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:36:32.426546 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.426466    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:36:32.426546 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.426478    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:36:32.426546 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.426495    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.427083 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.427069    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.427137 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.427094    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:36:32.427735 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.427721    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:36:32.427797 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.427748    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:36:32.427797 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.427759    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:36:32.443948 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.443808    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.444021 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.443953    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-161.us-east-2.compute.internal\": node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:32.458416 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.458393    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found" node="ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.460589 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.460573    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found" node="ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.469121 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.469108    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:32.556963 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.556939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.557066 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.556965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.557066 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.556982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/157678277e297843d51e22f1021ace90-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"157678277e297843d51e22f1021ace90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.569585 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.569563    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:32.658112 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.658083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.658244 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.658112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.658244 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.658139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/157678277e297843d51e22f1021ace90-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"157678277e297843d51e22f1021ace90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.658244 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.658194    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.658244 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.658194    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.658244 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.658192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/157678277e297843d51e22f1021ace90-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"157678277e297843d51e22f1021ace90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.670183 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.670163    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:32.761926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.761886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.764038 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:32.764024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:32.771035 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.771019    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:32.871541 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.871516    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:32.972064 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:32.972036    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:33.072552 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:33.072500    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:33.173338 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:33.173315    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:33.194510 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.194481    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 20:36:33.219584 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.219560    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:36:33.273741 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:33.273722    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:33.316937 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.316916    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:36:33.318640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.318624    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:36:33.318734 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.318658    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:36:33.318734 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.318673    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:36:33.374817 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:33.374787    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:33.386199 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.386174    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:36:33.418332 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.418263    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:31:32 +0000 UTC" deadline="2028-03-27 02:19:45.879084861 +0000 UTC"
Oct 30 20:36:33.418332 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.418330    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12317h43m12.460758069s"
Oct 30 20:36:33.453289 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:33.453259    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157678277e297843d51e22f1021ace90.slice/crio-fd6a2570855240aa73ed8dc4ce409b82a7a0eba2fc5762c15c3c61341c0921be WatchSource:0}: Error finding container fd6a2570855240aa73ed8dc4ce409b82a7a0eba2fc5762c15c3c61341c0921be: Status 404 returned error can't find the container with id fd6a2570855240aa73ed8dc4ce409b82a7a0eba2fc5762c15c3c61341c0921be
Oct 30 20:36:33.453526 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:33.453511    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2241538db95cc825a75346da6583a23.slice/crio-4d9649e08d287f9cf3755bdc1e0fd411c4abdfd438f4612f03ff2217c604f5c2 WatchSource:0}: Error finding container 4d9649e08d287f9cf3755bdc1e0fd411c4abdfd438f4612f03ff2217c604f5c2: Status 404 returned error can't find the container with id 4d9649e08d287f9cf3755bdc1e0fd411c4abdfd438f4612f03ff2217c604f5c2
Oct 30 20:36:33.459252 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.459170    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:36:33.475101 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:33.475081    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:36:33.543649 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.543634    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:36:33.555863 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.555841    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:33.570999 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.570972    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:36:33.571928 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.571917    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:36:33.580349 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.580331    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:36:33.632111 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:33.632064    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:36:34.233217 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.233176    2576 apiserver.go:52] "Watching apiserver"
Oct 30 20:36:34.240557 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.240533    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:36:34.240955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.240916    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-clqzg","openshift-image-registry/node-ca-sq5lx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-k4spg","openshift-cluster-node-tuning-operator/tuned-bkx7g","openshift-dns/node-resolver-4mv46","openshift-multus/network-metrics-daemon-gtkw8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs","openshift-network-diagnostics/network-check-target-lfl79","openshift-network-operator/iptables-alerter-nm42c","kube-system/global-pull-secret-syncer-vmcq2","kube-system/konnectivity-agent-vgcxk","kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-88xhl"]
Oct 30 20:36:34.243378 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.243356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:34.243470 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.243452    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:34.245738 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.245716    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:34.245869 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.245778    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:34.249361 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.248982    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.250133 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.250113    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.252329 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.252305    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.254556 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.254535    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 20:36:34.254984 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.254962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.257025 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.256989    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:36:34.257025 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.257014    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x97dd\""
Oct 30 20:36:34.257164 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.257105    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-82bc7\""
Oct 30 20:36:34.257215 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.257197    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:36:34.257493 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.257255    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fxgqk\""
Oct 30 20:36:34.257493 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.257281    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:36:34.257493 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.257333    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:36:34.257493 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.257349    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:36:34.257493 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.257443    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:34.257738 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.257491    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:34.258007 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.257987    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:36:34.258084 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.258059    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:36:34.258141 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.258133    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:36:34.258230 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.258216    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:36:34.258302 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.258282    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:36:34.258358 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.258331    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:36:34.258358 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.258354    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:36:34.258455 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.258379    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:36:34.258455 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.258428    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:36:34.258554 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.258493    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:36:34.259676 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.259643    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l4rxj\""
Oct 30 20:36:34.260917 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.260896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.263687 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.263666    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.267260 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.267241    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.267801 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.267781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-cnibin\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.267916 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.267810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-systemd\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.267916 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.267825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-tmp\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.267916 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.267842    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-netd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.267916 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.267876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.267916 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.267903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8mz\" (UniqueName: \"kubernetes.io/projected/e30414b5-8768-4eb1-a4ba-395115dba647-kube-api-access-cj8mz\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.268157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.267926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxvc\" (UniqueName: \"kubernetes.io/projected/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-kube-api-access-qnxvc\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:34.268157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.267979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.268157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-sys\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.268157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-lib-modules\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.268157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-host\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.268330 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-kubelet\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.268330 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-ovn\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.268330 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysconfig\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.268330 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-conf\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.268439 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-var-lib-kubelet\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.268439 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-slash\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.268571 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-log-socket\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.268635 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-config\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.268635 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-script-lib\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.268697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-system-cni-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.268697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-run\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.268697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-etc-tuned\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.268697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-var-lib-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.268822 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-etc-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.268822 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-bin\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.268822 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268783    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7389168a-7f19-48df-98f8-6d6d20892416-hosts-file\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.268822 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.269024 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xph2w\" (UniqueName: \"kubernetes.io/projected/8902c3bd-bf64-48e1-b113-d42ed8456348-kube-api-access-xph2w\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.269024 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-kubernetes\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.269024 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qgww\" (UniqueName: \"kubernetes.io/projected/2ba0647d-e918-4bc7-843b-205e600536ec-kube-api-access-2qgww\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.269024 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268960    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:34.269024 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.268982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-dbus\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:34.269024 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-netns\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.269267 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-systemd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.269267 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-kubelet-config\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:34.269267 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-binary-copy\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.269267 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:34.269267 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.269267 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.269267 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-env-overrides\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.269267 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bv5x\" (UniqueName: \"kubernetes.io/projected/7389168a-7f19-48df-98f8-6d6d20892416-kube-api-access-2bv5x\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.269267 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.269662 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.269662 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:34.269662 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-node-log\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.269662 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e30414b5-8768-4eb1-a4ba-395115dba647-ovn-node-metrics-cert\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.269662 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-modprobe-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.269662 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269393    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-systemd-units\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.269662 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7389168a-7f19-48df-98f8-6d6d20892416-tmp-dir\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.269975 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.269844    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.271942 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.271925    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:36:34.272280 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.272253    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:36:34.303031 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.303000    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9gwql"
Oct 30 20:36:34.314275 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.314259    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9gwql"
Oct 30 20:36:34.325138 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.325093    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal" event={"ID":"157678277e297843d51e22f1021ace90","Type":"ContainerStarted","Data":"fd6a2570855240aa73ed8dc4ce409b82a7a0eba2fc5762c15c3c61341c0921be"}
Oct 30 20:36:34.326526 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.326482    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal" event={"ID":"e2241538db95cc825a75346da6583a23","Type":"ContainerStarted","Data":"4d9649e08d287f9cf3755bdc1e0fd411c4abdfd438f4612f03ff2217c604f5c2"}
Oct 30 20:36:34.357070 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.357049    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:36:34.365925 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.365636    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:36:34.365925 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.365652    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7lq4m\""
Oct 30 20:36:34.365925 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.365687    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:36:34.365925 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.365739    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bcgjk\""
Oct 30 20:36:34.365925 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.365770    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:36:34.365925 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.365787    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:36:34.365925 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.365875    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-chh5h\""
Oct 30 20:36:34.366188 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.365985    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:36:34.366188 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.366068    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:36:34.366188 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.366160    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:36:34.366188 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.366169    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:36:34.366188 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.365875    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6lk59\""
Oct 30 20:36:34.366352 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.366218    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:36:34.366352 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.366224    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2hx48\""
Oct 30 20:36:34.366431 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.366370    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:36:34.366431 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.366387    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:36:34.366904 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.366840    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:36:34.370238 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-system-cni-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.370326 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-run\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.370326 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370281    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-system-cni-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.370326 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xph2w\" (UniqueName: \"kubernetes.io/projected/8902c3bd-bf64-48e1-b113-d42ed8456348-kube-api-access-xph2w\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.370326 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgww\" (UniqueName: \"kubernetes.io/projected/2ba0647d-e918-4bc7-843b-205e600536ec-kube-api-access-2qgww\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.370507 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-dbus\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:34.370507 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-netns\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.370507 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-systemd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.370507 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.370507 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.370507 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-kubelet-config\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:34.370507 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-binary-copy\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.370507 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370496    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-systemd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370529    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-dbus\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58b187ea-dead-4a37-8a52-5e447783d815-serviceca\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370575    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-kubelet-config\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e30414b5-8768-4eb1-a4ba-395115dba647-ovn-node-metrics-cert\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370622    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-etc-kubernetes\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-registration-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-systemd-units\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370456    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-run\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370549    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-netns\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-etc-selinux\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-cnibin\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.370819 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-netd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8mz\" (UniqueName: \"kubernetes.io/projected/e30414b5-8768-4eb1-a4ba-395115dba647-kube-api-access-cj8mz\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370923    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-cnibin\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.370936    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cni-binary-copy\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371049    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-sys-fs\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-systemd-units\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.371018    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371063    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-netd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxvc\" (UniqueName: \"kubernetes.io/projected/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-kube-api-access-qnxvc\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.371115    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.371126    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret podName:a248c2d8-ba41-4190-9ca7-6cce702dd07a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:34.871106501 +0000 UTC m=+3.006629653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret") pod "global-pull-secret-syncer-vmcq2" (UID: "a248c2d8-ba41-4190-9ca7-6cce702dd07a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.371158    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs podName:3efcea4c-b6b6-47a5-be1d-c61a83bbe907 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:34.871143608 +0000 UTC m=+3.006666746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs") pod "network-metrics-daemon-gtkw8" (UID: "3efcea4c-b6b6-47a5-be1d-c61a83bbe907") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-sys\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.371514 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371354    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-sys\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-kubelet\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371452    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-kubelet\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371458    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371479    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zlkc\" (UniqueName: \"kubernetes.io/projected/73fa532a-7546-4e60-8f8c-901cebfd7f07-kube-api-access-7zlkc\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2a1e731-e195-4a0b-b778-c606ccc13ef6-host-slash\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysconfig\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-log-socket\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-config\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371578    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7389168a-7f19-48df-98f8-6d6d20892416-hosts-file\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371624    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysconfig\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-device-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-etc-tuned\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-var-lib-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371720    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-log-socket\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-etc-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.372243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-bin\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-kubelet\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371768    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7389168a-7f19-48df-98f8-6d6d20892416-hosts-file\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-kubernetes\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.371974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-env-overrides\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-socket-dir-parent\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372070    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-var-lib-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-kubernetes\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372121    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-bin\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372167    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-etc-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-bin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzzjh\" (UniqueName: \"kubernetes.io/projected/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-kube-api-access-fzzjh\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bv5x\" (UniqueName: \"kubernetes.io/projected/7389168a-7f19-48df-98f8-6d6d20892416-kube-api-access-2bv5x\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-system-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.373090 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-multus-certs\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372359    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cwvl\" (UniqueName: \"kubernetes.io/projected/b2a1e731-e195-4a0b-b778-c606ccc13ef6-kube-api-access-7cwvl\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-node-log\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372413    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-env-overrides\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-daemon-config\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372520    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372520    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372517    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-config\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51d19e04-9e59-4f4f-be5c-9900cca6d655-agent-certs\") pod \"konnectivity-agent-vgcxk\" (UID: \"51d19e04-9e59-4f4f-be5c-9900cca6d655\") " pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372570    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-node-log\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-modprobe-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cnibin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372727    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-modprobe-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-multus\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-hostroot\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372888    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.373927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372895    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-socket-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-systemd\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372979    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-binary-copy\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.372994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-tmp\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-systemd\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-netns\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58b187ea-dead-4a37-8a52-5e447783d815-host\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpbbp\" (UniqueName: \"kubernetes.io/projected/58b187ea-dead-4a37-8a52-5e447783d815-kube-api-access-gpbbp\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51d19e04-9e59-4f4f-be5c-9900cca6d655-konnectivity-ca\") pod \"konnectivity-agent-vgcxk\" (UID: \"51d19e04-9e59-4f4f-be5c-9900cca6d655\") " pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7389168a-7f19-48df-98f8-6d6d20892416-tmp-dir\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-lib-modules\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-host\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-ovn\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373304    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-script-lib\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-k8s-cni-cncf-io\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-conf-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.374477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373360    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-lib-modules\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-conf\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-ovn\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-var-lib-kubelet\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373475    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-var-lib-kubelet\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373479    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7389168a-7f19-48df-98f8-6d6d20892416-tmp-dir\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-slash\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2a1e731-e195-4a0b-b778-c606ccc13ef6-iptables-alerter-script\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373626    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-slash\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373648    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-conf\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373665    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-host\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373689    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.375157 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.373845    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-script-lib\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.375621 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.375470    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-etc-tuned\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.375621 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.375488    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-tmp\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.375714 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.375689    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e30414b5-8768-4eb1-a4ba-395115dba647-ovn-node-metrics-cert\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.382029 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.382007    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:36:34.382029 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.382031    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:36:34.382149 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.382044    2576 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksf for pod openshift-network-diagnostics/network-check-target-lfl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:34.382149 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.382088    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf podName:2fc2fd6c-3d77-4446-9402-cf73cdd3497a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:34.882078266 +0000 UTC m=+3.017601430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksf" (UniqueName: "kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf") pod "network-check-target-lfl79" (UID: "2fc2fd6c-3d77-4446-9402-cf73cdd3497a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:34.383294 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.383270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xph2w\" (UniqueName: \"kubernetes.io/projected/8902c3bd-bf64-48e1-b113-d42ed8456348-kube-api-access-xph2w\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.384572 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.384546    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxvc\" (UniqueName: \"kubernetes.io/projected/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-kube-api-access-qnxvc\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:34.385010 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.384990    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qgww\" (UniqueName: \"kubernetes.io/projected/2ba0647d-e918-4bc7-843b-205e600536ec-kube-api-access-2qgww\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.385529 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.385468    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bv5x\" (UniqueName: \"kubernetes.io/projected/7389168a-7f19-48df-98f8-6d6d20892416-kube-api-access-2bv5x\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.386816 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.386795    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8mz\" (UniqueName: \"kubernetes.io/projected/e30414b5-8768-4eb1-a4ba-395115dba647-kube-api-access-cj8mz\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.474496 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlkc\" (UniqueName: \"kubernetes.io/projected/73fa532a-7546-4e60-8f8c-901cebfd7f07-kube-api-access-7zlkc\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.474496 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2a1e731-e195-4a0b-b778-c606ccc13ef6-host-slash\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.474697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-device-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.474697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-kubelet\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474587    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-socket-dir-parent\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-bin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzzjh\" (UniqueName: \"kubernetes.io/projected/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-kube-api-access-fzzjh\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-system-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474697 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-multus-certs\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cwvl\" (UniqueName: \"kubernetes.io/projected/b2a1e731-e195-4a0b-b778-c606ccc13ef6-kube-api-access-7cwvl\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.474955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-daemon-config\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51d19e04-9e59-4f4f-be5c-9900cca6d655-agent-certs\") pod \"konnectivity-agent-vgcxk\" (UID: \"51d19e04-9e59-4f4f-be5c-9900cca6d655\") " pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:36:34.474955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cnibin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-multus\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-hostroot\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-socket-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.474955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-netns\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.474955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58b187ea-dead-4a37-8a52-5e447783d815-host\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.474955 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbbp\" (UniqueName: \"kubernetes.io/projected/58b187ea-dead-4a37-8a52-5e447783d815-kube-api-access-gpbbp\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.474972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51d19e04-9e59-4f4f-be5c-9900cca6d655-konnectivity-ca\") pod \"konnectivity-agent-vgcxk\" (UID: \"51d19e04-9e59-4f4f-be5c-9900cca6d655\") " pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-k8s-cni-cncf-io\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-conf-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2a1e731-e195-4a0b-b778-c606ccc13ef6-iptables-alerter-script\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58b187ea-dead-4a37-8a52-5e447783d815-serviceca\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-etc-kubernetes\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475162    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-registration-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-etc-selinux\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cni-binary-copy\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.475313 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-sys-fs\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.475765 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475330    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-sys-fs\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.475765 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475520    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2a1e731-e195-4a0b-b778-c606ccc13ef6-host-slash\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.475765 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475567    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-device-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.475765 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475602    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-kubelet\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.475765 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475648    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-socket-dir-parent\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.475765 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475683    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-bin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.476004 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-system-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.476004 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.475968    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-multus-certs\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.476398 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476371    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-k8s-cni-cncf-io\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.476483 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476448    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-etc-kubernetes\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.476483 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476468    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58b187ea-dead-4a37-8a52-5e447783d815-host\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.476606 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476571    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-conf-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.476696 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-registration-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.476807 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476675    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-etc-selinux\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.476807 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476713    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.476807 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476725    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.476807 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476773    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-multus\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.476807 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476775    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-hostroot\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.476807 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476770    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-netns\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.477126 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.476932    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cnibin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.477126 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.477065    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-socket-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.477378 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.477355    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-daemon-config\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.477664 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.477647    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2a1e731-e195-4a0b-b778-c606ccc13ef6-iptables-alerter-script\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.477744 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.477725    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cni-binary-copy\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.477806 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.477794    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58b187ea-dead-4a37-8a52-5e447783d815-serviceca\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.477884 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.477871    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51d19e04-9e59-4f4f-be5c-9900cca6d655-konnectivity-ca\") pod \"konnectivity-agent-vgcxk\" (UID: \"51d19e04-9e59-4f4f-be5c-9900cca6d655\") " pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:36:34.480177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.480160    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51d19e04-9e59-4f4f-be5c-9900cca6d655-agent-certs\") pod \"konnectivity-agent-vgcxk\" (UID: \"51d19e04-9e59-4f4f-be5c-9900cca6d655\") " pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:36:34.490744 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.490599    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbbp\" (UniqueName: \"kubernetes.io/projected/58b187ea-dead-4a37-8a52-5e447783d815-kube-api-access-gpbbp\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.490744 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.490698    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cwvl\" (UniqueName: \"kubernetes.io/projected/b2a1e731-e195-4a0b-b778-c606ccc13ef6-kube-api-access-7cwvl\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.492332 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.491567    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zlkc\" (UniqueName: \"kubernetes.io/projected/73fa532a-7546-4e60-8f8c-901cebfd7f07-kube-api-access-7zlkc\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.492332 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.491982    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzzjh\" (UniqueName: \"kubernetes.io/projected/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-kube-api-access-fzzjh\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.586063 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.586023    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:36:34.588886 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.588550    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:36:34.592011 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.591987    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:36:34.600202 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:34.600139    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba0647d_e918_4bc7_843b_205e600536ec.slice/crio-2bc375e65aa0ea5c65e459c525e8799fb69e57e2e8ac0f505f75ad990d57585a WatchSource:0}: Error finding container 2bc375e65aa0ea5c65e459c525e8799fb69e57e2e8ac0f505f75ad990d57585a: Status 404 returned error can't find the container with id 2bc375e65aa0ea5c65e459c525e8799fb69e57e2e8ac0f505f75ad990d57585a
Oct 30 20:36:34.602947 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:34.602924    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7389168a_7f19_48df_98f8_6d6d20892416.slice/crio-450407d6c8501801d763d37fa3ef48d218689f995b03dc5ed8bc4fce95a3e554 WatchSource:0}: Error finding container 450407d6c8501801d763d37fa3ef48d218689f995b03dc5ed8bc4fce95a3e554: Status 404 returned error can't find the container with id 450407d6c8501801d763d37fa3ef48d218689f995b03dc5ed8bc4fce95a3e554
Oct 30 20:36:34.608645 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.608616    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:34.610740 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.610720    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:36:34.613480 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.613462    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:36:34.618923 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:34.618901    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30414b5_8768_4eb1_a4ba_395115dba647.slice/crio-d5f43122589f19a195e82f24618908f0f05878856b1a3e1d4bb2f18609ecfc7d WatchSource:0}: Error finding container d5f43122589f19a195e82f24618908f0f05878856b1a3e1d4bb2f18609ecfc7d: Status 404 returned error can't find the container with id d5f43122589f19a195e82f24618908f0f05878856b1a3e1d4bb2f18609ecfc7d
Oct 30 20:36:34.620430 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.620274    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:36:34.626468 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:34.626445    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b187ea_dead_4a37_8a52_5e447783d815.slice/crio-8463e4ae87cc0710ca9aad3fcffda20a840c5a06db2a0c113445af48fe880008 WatchSource:0}: Error finding container 8463e4ae87cc0710ca9aad3fcffda20a840c5a06db2a0c113445af48fe880008: Status 404 returned error can't find the container with id 8463e4ae87cc0710ca9aad3fcffda20a840c5a06db2a0c113445af48fe880008
Oct 30 20:36:34.627989 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.627969    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-clqzg"
Oct 30 20:36:34.633243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.633223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:36:34.633609 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:34.633586    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a1e731_e195_4a0b_b778_c606ccc13ef6.slice/crio-1dad5510556bc61a7a7c6d5b3110720bcfe7b95fc35c5038fa78e3c1d72256a4 WatchSource:0}: Error finding container 1dad5510556bc61a7a7c6d5b3110720bcfe7b95fc35c5038fa78e3c1d72256a4: Status 404 returned error can't find the container with id 1dad5510556bc61a7a7c6d5b3110720bcfe7b95fc35c5038fa78e3c1d72256a4
Oct 30 20:36:34.642812 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:34.642789    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d37e01_bf52_40e0_bc5a_524ce20ddad4.slice/crio-d1e310a1152b31ec7ee355d51efa554069bcd8a6fb1c07c58982aa4264b9b55a WatchSource:0}: Error finding container d1e310a1152b31ec7ee355d51efa554069bcd8a6fb1c07c58982aa4264b9b55a: Status 404 returned error can't find the container with id d1e310a1152b31ec7ee355d51efa554069bcd8a6fb1c07c58982aa4264b9b55a
Oct 30 20:36:34.643872 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:36:34.643830    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d19e04_9e59_4f4f_be5c_9900cca6d655.slice/crio-e0c8d484c3330acc4ec03058786ab4f68a8cd5fbde42e9da92f9bf28e19f3ec6 WatchSource:0}: Error finding container e0c8d484c3330acc4ec03058786ab4f68a8cd5fbde42e9da92f9bf28e19f3ec6: Status 404 returned error can't find the container with id e0c8d484c3330acc4ec03058786ab4f68a8cd5fbde42e9da92f9bf28e19f3ec6
Oct 30 20:36:34.877964 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.877894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:34.877964 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.877936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:34.878124 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.878050    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:34.878124 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.878099    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret podName:a248c2d8-ba41-4190-9ca7-6cce702dd07a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:35.87808447 +0000 UTC m=+4.013607602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret") pod "global-pull-secret-syncer-vmcq2" (UID: "a248c2d8-ba41-4190-9ca7-6cce702dd07a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:34.878184 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.878049    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:34.878184 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.878174    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs podName:3efcea4c-b6b6-47a5-be1d-c61a83bbe907 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:35.878161294 +0000 UTC m=+4.013684427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs") pod "network-metrics-daemon-gtkw8" (UID: "3efcea4c-b6b6-47a5-be1d-c61a83bbe907") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:34.978913 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:34.978422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:34.978913 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.978575    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:36:34.978913 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.978593    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:36:34.978913 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.978605    2576 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksf for pod openshift-network-diagnostics/network-check-target-lfl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:34.978913 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:34.978663    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf podName:2fc2fd6c-3d77-4446-9402-cf73cdd3497a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:35.978645323 +0000 UTC m=+4.114168468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksf" (UniqueName: "kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf") pod "network-check-target-lfl79" (UID: "2fc2fd6c-3d77-4446-9402-cf73cdd3497a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:35.317369 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.317059    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:31:34 +0000 UTC" deadline="2028-05-08 08:46:03.647550968 +0000 UTC"
Oct 30 20:36:35.317369 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.317316    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13332h9m28.330240277s"
Oct 30 20:36:35.321646 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.321174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:35.321646 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:35.321296    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:35.336185 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.336147    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-clqzg" event={"ID":"d8d37e01-bf52-40e0-bc5a-524ce20ddad4","Type":"ContainerStarted","Data":"d1e310a1152b31ec7ee355d51efa554069bcd8a6fb1c07c58982aa4264b9b55a"}
Oct 30 20:36:35.337794 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.337746    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sq5lx" event={"ID":"58b187ea-dead-4a37-8a52-5e447783d815","Type":"ContainerStarted","Data":"8463e4ae87cc0710ca9aad3fcffda20a840c5a06db2a0c113445af48fe880008"}
Oct 30 20:36:35.345756 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.345732    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs" event={"ID":"73fa532a-7546-4e60-8f8c-901cebfd7f07","Type":"ContainerStarted","Data":"27c3a01f6e08f36046a0ba85282af82e507a7c028a684a99bdd2e8c672e8a2d7"}
Oct 30 20:36:35.348793 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.348699    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg" event={"ID":"e30414b5-8768-4eb1-a4ba-395115dba647","Type":"ContainerStarted","Data":"d5f43122589f19a195e82f24618908f0f05878856b1a3e1d4bb2f18609ecfc7d"}
Oct 30 20:36:35.354657 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.354035    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal" event={"ID":"157678277e297843d51e22f1021ace90","Type":"ContainerStarted","Data":"1f5a051a5f80989fbea5ad8634938a19b153873f0145352bc86d7efa771f526a"}
Oct 30 20:36:35.359949 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.359327    2576 generic.go:356] "Generic (PLEG): container finished" podID="e2241538db95cc825a75346da6583a23" containerID="c117e6fa0635cae5410bf81eda29098b73d71d894b45bfa987370e8056af0879" exitCode=0
Oct 30 20:36:35.359949 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.359395    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal" event={"ID":"e2241538db95cc825a75346da6583a23","Type":"ContainerDied","Data":"c117e6fa0635cae5410bf81eda29098b73d71d894b45bfa987370e8056af0879"}
Oct 30 20:36:35.369391 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.369365    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nm42c" event={"ID":"b2a1e731-e195-4a0b-b778-c606ccc13ef6","Type":"ContainerStarted","Data":"1dad5510556bc61a7a7c6d5b3110720bcfe7b95fc35c5038fa78e3c1d72256a4"}
Oct 30 20:36:35.378361 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.378282    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4mv46" event={"ID":"7389168a-7f19-48df-98f8-6d6d20892416","Type":"ContainerStarted","Data":"450407d6c8501801d763d37fa3ef48d218689f995b03dc5ed8bc4fce95a3e554"}
Oct 30 20:36:35.383056 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.383032    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkx7g" event={"ID":"2ba0647d-e918-4bc7-843b-205e600536ec","Type":"ContainerStarted","Data":"2bc375e65aa0ea5c65e459c525e8799fb69e57e2e8ac0f505f75ad990d57585a"}
Oct 30 20:36:35.384960 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.384925    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88xhl" event={"ID":"8902c3bd-bf64-48e1-b113-d42ed8456348","Type":"ContainerStarted","Data":"f326d3df10a2dc09503dae8b594c653b2397e25df090e25d839dac4162fb7ef9"}
Oct 30 20:36:35.387587 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.387561    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgcxk" event={"ID":"51d19e04-9e59-4f4f-be5c-9900cca6d655","Type":"ContainerStarted","Data":"e0c8d484c3330acc4ec03058786ab4f68a8cd5fbde42e9da92f9bf28e19f3ec6"}
Oct 30 20:36:35.394733 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.394687    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal" podStartSLOduration=2.394673545 podStartE2EDuration="2.394673545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:35.371135851 +0000 UTC m=+3.506659006" watchObservedRunningTime="2026-10-30 20:36:35.394673545 +0000 UTC m=+3.530196715"
Oct 30 20:36:35.886164 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.885404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:35.886164 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.885451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:35.886164 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:35.885609    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:35.886164 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:35.885667    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret podName:a248c2d8-ba41-4190-9ca7-6cce702dd07a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:37.885649261 +0000 UTC m=+6.021172409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret") pod "global-pull-secret-syncer-vmcq2" (UID: "a248c2d8-ba41-4190-9ca7-6cce702dd07a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:35.886164 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:35.886050    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:35.886164 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:35.886098    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs podName:3efcea4c-b6b6-47a5-be1d-c61a83bbe907 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:37.886084218 +0000 UTC m=+6.021607364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs") pod "network-metrics-daemon-gtkw8" (UID: "3efcea4c-b6b6-47a5-be1d-c61a83bbe907") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:35.986756 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:35.986722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:35.986975 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:35.986952    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:36:35.987090 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:35.986985    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:36:35.987090 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:35.986997    2576 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksf for pod openshift-network-diagnostics/network-check-target-lfl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:35.987090 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:35.987049    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf podName:2fc2fd6c-3d77-4446-9402-cf73cdd3497a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:37.987030908 +0000 UTC m=+6.122554040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksf" (UniqueName: "kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf") pod "network-check-target-lfl79" (UID: "2fc2fd6c-3d77-4446-9402-cf73cdd3497a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:36.337086 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:36.336998    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:31:34 +0000 UTC" deadline="2028-07-24 05:53:05.839161598 +0000 UTC"
Oct 30 20:36:36.337086 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:36.337035    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15177h16m29.50213103s"
Oct 30 20:36:36.337563 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:36.337222    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:36.337563 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:36.337352    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:36.338038 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:36.337663    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:36.338038 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:36.337736    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:36.338038 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:36.337903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:36.338038 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:36.337988    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:36.395685 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:36.395654    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal" event={"ID":"e2241538db95cc825a75346da6583a23","Type":"ContainerStarted","Data":"a1b4a59f50b719362f249dbbfe1fd01733c5884406e210b5899175bd88263257"}
Oct 30 20:36:37.904590 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:37.904513    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:37.904590 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:37.904566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:37.905102 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:37.904710    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:37.905102 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:37.904767    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret podName:a248c2d8-ba41-4190-9ca7-6cce702dd07a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:41.904750564 +0000 UTC m=+10.040273721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret") pod "global-pull-secret-syncer-vmcq2" (UID: "a248c2d8-ba41-4190-9ca7-6cce702dd07a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:37.905213 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:37.905179    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:37.905255 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:37.905227    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs podName:3efcea4c-b6b6-47a5-be1d-c61a83bbe907 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:41.905211662 +0000 UTC m=+10.040734808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs") pod "network-metrics-daemon-gtkw8" (UID: "3efcea4c-b6b6-47a5-be1d-c61a83bbe907") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:38.005490 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:38.004895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:38.005490 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:38.005063    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:36:38.005490 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:38.005081    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:36:38.005490 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:38.005092    2576 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksf for pod openshift-network-diagnostics/network-check-target-lfl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:38.005490 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:38.005140    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf podName:2fc2fd6c-3d77-4446-9402-cf73cdd3497a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:42.005124333 +0000 UTC m=+10.140647478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksf" (UniqueName: "kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf") pod "network-check-target-lfl79" (UID: "2fc2fd6c-3d77-4446-9402-cf73cdd3497a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:38.324649 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:38.323990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:38.324649 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:38.324131    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:38.324649 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:38.324338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:38.324649 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:38.324432    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:38.324649 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:38.324489    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:38.324649 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:38.324561    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:38.402622 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:38.402593    2576 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="9c4777506e01804a8ad316b1f2133c5d65ad9cb5230d922abbb3aef9d7b9ce73" exitCode=0
Oct 30 20:36:38.402725 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:38.402651    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88xhl" event={"ID":"8902c3bd-bf64-48e1-b113-d42ed8456348","Type":"ContainerDied","Data":"9c4777506e01804a8ad316b1f2133c5d65ad9cb5230d922abbb3aef9d7b9ce73"}
Oct 30 20:36:38.429165 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:38.429087    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal" podStartSLOduration=5.429071814 podStartE2EDuration="5.429071814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:36.412028262 +0000 UTC m=+4.547551398" watchObservedRunningTime="2026-10-30 20:36:38.429071814 +0000 UTC m=+6.564594970"
Oct 30 20:36:39.408834 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:39.408213    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgcxk" event={"ID":"51d19e04-9e59-4f4f-be5c-9900cca6d655","Type":"ContainerStarted","Data":"c9d0bbb5a6342cce5c6f60ced374f515a0b365fc079e664cdcb5b4c955194ed2"}
Oct 30 20:36:40.324361 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:40.324328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:40.324504 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:40.324453    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:40.324830 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:40.324808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:40.324971 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:40.324936    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:40.325035 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:40.325002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:40.325092 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:40.325066    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:40.413433 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:40.413344    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sq5lx" event={"ID":"58b187ea-dead-4a37-8a52-5e447783d815","Type":"ContainerStarted","Data":"31ecfe7a77a96bab37e5a77850ee7051ab58ddd677fe216bf9d8081d16f8adb5"}
Oct 30 20:36:40.431116 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:40.431068    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vgcxk" podStartSLOduration=4.690830025 podStartE2EDuration="8.431050675s" totalImagesPullingTime="3.74022065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:39.425497587 +0000 UTC m=+7.561020741" watchObservedRunningTime="2026-10-30 20:36:40.431050675 +0000 UTC m=+8.566573830"
Oct 30 20:36:40.431411 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:40.431383    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sq5lx" podStartSLOduration=3.719682147 podStartE2EDuration="8.431375527s" totalImagesPullingTime="4.71169338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:40.430565633 +0000 UTC m=+8.566088790" watchObservedRunningTime="2026-10-30 20:36:40.431375527 +0000 UTC m=+8.566898683"
Oct 30 20:36:41.421797 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:41.421686    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs" event={"ID":"73fa532a-7546-4e60-8f8c-901cebfd7f07","Type":"ContainerStarted","Data":"78ac98452cbce777a4cd17fb4fdf7597db28c87af6148bdc35cf7a4a6d307166"}
Oct 30 20:36:41.940483 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:41.940444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:41.940672 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:41.940499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:41.940672 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:41.940665    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:41.940786 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:41.940727    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret podName:a248c2d8-ba41-4190-9ca7-6cce702dd07a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:49.940709777 +0000 UTC m=+18.076232929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret") pod "global-pull-secret-syncer-vmcq2" (UID: "a248c2d8-ba41-4190-9ca7-6cce702dd07a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:41.940983 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:41.940969    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:41.941056 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:41.941019    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs podName:3efcea4c-b6b6-47a5-be1d-c61a83bbe907 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:49.941006378 +0000 UTC m=+18.076529531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs") pod "network-metrics-daemon-gtkw8" (UID: "3efcea4c-b6b6-47a5-be1d-c61a83bbe907") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:42.042014 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.041389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:42.042014 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:42.041577    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:36:42.042014 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:42.041596    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:36:42.042014 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:42.041608    2576 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksf for pod openshift-network-diagnostics/network-check-target-lfl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:42.042014 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:42.041664    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf podName:2fc2fd6c-3d77-4446-9402-cf73cdd3497a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:50.041647074 +0000 UTC m=+18.177170222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksf" (UniqueName: "kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf") pod "network-check-target-lfl79" (UID: "2fc2fd6c-3d77-4446-9402-cf73cdd3497a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:42.257688 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.257065    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:36:42.323884 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.322980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:42.323884 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:42.323108    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:42.323884 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.323510    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:42.323884 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:42.323625    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:42.323884 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.323750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:42.323884 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:42.323833    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:42.341243 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.340969    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:36:42.257084015Z","UUID":"97a25352-3301-4367-83f1-6b84c7a7a944","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:36:42.342684 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.342664    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:36:42.342788 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.342695    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:36:42.425752 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.425192    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4mv46" event={"ID":"7389168a-7f19-48df-98f8-6d6d20892416","Type":"ContainerStarted","Data":"9f7a61b4d4021775490463bfca538592863524400d29c98afa26efcf26884f01"}
Oct 30 20:36:42.427805 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.427423    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-clqzg" event={"ID":"d8d37e01-bf52-40e0-bc5a-524ce20ddad4","Type":"ContainerStarted","Data":"d9f27321395ebabe81226715def29a63d75bee0b0c3a1d9fbc8c81a04782e793"}
Oct 30 20:36:42.430755 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.430697    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs" event={"ID":"73fa532a-7546-4e60-8f8c-901cebfd7f07","Type":"ContainerStarted","Data":"d7ee42cd164a9c09b92f7c9910f0a6aa64c0a6f0319199cfbc8a242f830216fc"}
Oct 30 20:36:42.457693 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.457606    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-clqzg" podStartSLOduration=2.999927492 podStartE2EDuration="10.457591866s" totalImagesPullingTime="7.457664374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:42.457326965 +0000 UTC m=+10.592850118" watchObservedRunningTime="2026-10-30 20:36:42.457591866 +0000 UTC m=+10.593115020"
Oct 30 20:36:42.458130 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:42.458095    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4mv46" podStartSLOduration=2.8549073 podStartE2EDuration="10.45808434s" totalImagesPullingTime="7.60317704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:42.44129644 +0000 UTC m=+10.576819573" watchObservedRunningTime="2026-10-30 20:36:42.45808434 +0000 UTC m=+10.593607495"
Oct 30 20:36:43.435184 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:43.435027    2576 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="dd9a7a856055be0b7ad43dcb8223c7222afe1ad7bce172d72f8450159414e9e7" exitCode=0
Oct 30 20:36:43.435536 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:43.435191    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88xhl" event={"ID":"8902c3bd-bf64-48e1-b113-d42ed8456348","Type":"ContainerDied","Data":"dd9a7a856055be0b7ad43dcb8223c7222afe1ad7bce172d72f8450159414e9e7"}
Oct 30 20:36:44.122450 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:44.122376    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:36:44.123092 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:44.122932    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:36:44.321498 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:44.321468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:44.321498 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:44.321532    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:44.321722 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:44.321540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:44.321822 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:44.321718    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:44.321822 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:44.321775    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:44.321952 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:44.321886    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:44.438709 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:44.438610    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nm42c" event={"ID":"b2a1e731-e195-4a0b-b778-c606ccc13ef6","Type":"ContainerStarted","Data":"4806a4346dd9fa0b2ee5f36bc67b99588809081b8417ca448726d170b6799635"}
Oct 30 20:36:44.440797 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:44.440768    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs" event={"ID":"73fa532a-7546-4e60-8f8c-901cebfd7f07","Type":"ContainerStarted","Data":"d11a58ce17ed4f85abb4b6047617c236662b6a40b917cf92f7dbcfbf1b4091cf"}
Oct 30 20:36:44.455377 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:44.455341    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nm42c" podStartSLOduration=4.792195142 podStartE2EDuration="12.455330055s" totalImagesPullingTime="7.663134913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:44.455015428 +0000 UTC m=+12.590538576" watchObservedRunningTime="2026-10-30 20:36:44.455330055 +0000 UTC m=+12.590853209"
Oct 30 20:36:44.475228 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:44.475189    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs" podStartSLOduration=4.203324225 podStartE2EDuration="12.4751835s" totalImagesPullingTime="8.271859275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:44.474416983 +0000 UTC m=+12.609940148" watchObservedRunningTime="2026-10-30 20:36:44.4751835 +0000 UTC m=+12.610706654"
Oct 30 20:36:45.444712 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:45.444672    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkx7g" event={"ID":"2ba0647d-e918-4bc7-843b-205e600536ec","Type":"ContainerStarted","Data":"49aaf0f3132a6b12d68f6b833a31f6d400c6875c4007dd44724871646299968b"}
Oct 30 20:36:45.447115 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:45.447087    2576 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="5d53003e0cf235e30f997c835c6344e1038e533f7abb650d4804ca7471068e54" exitCode=0
Oct 30 20:36:45.447258 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:45.447172    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88xhl" event={"ID":"8902c3bd-bf64-48e1-b113-d42ed8456348","Type":"ContainerDied","Data":"5d53003e0cf235e30f997c835c6344e1038e533f7abb650d4804ca7471068e54"}
Oct 30 20:36:45.462280 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:45.462236    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bkx7g" podStartSLOduration=3.4721277 podStartE2EDuration="13.462225614s" totalImagesPullingTime="9.990097914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:45.461519571 +0000 UTC m=+13.597042722" watchObservedRunningTime="2026-10-30 20:36:45.462225614 +0000 UTC m=+13.597748768"
Oct 30 20:36:46.322254 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:46.322140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:46.322254 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:46.322211    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:46.322254 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:46.322230    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:46.322502 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:46.322343    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:46.322502 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:46.322392    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:46.322582 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:46.322506    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:47.454439 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:47.454406    2576 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="bf64e111bc2eb938f9a13944a76f6a793a425f99148ab72b39ea6179ae73b26c" exitCode=0
Oct 30 20:36:47.455010 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:47.454451    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88xhl" event={"ID":"8902c3bd-bf64-48e1-b113-d42ed8456348","Type":"ContainerDied","Data":"bf64e111bc2eb938f9a13944a76f6a793a425f99148ab72b39ea6179ae73b26c"}
Oct 30 20:36:48.322947 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:48.321214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:48.322947 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:48.321234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:48.322947 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:48.321352    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:48.322947 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:48.321455    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:48.322947 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:48.321516    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:48.322947 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:48.321591    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:49.461748 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:49.461522    2576 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="dd6da7b9b0c6b3c90ae47d10799617ef6acc1dc11708d35296dd689550a2655e" exitCode=0
Oct 30 20:36:49.461748 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:49.461610    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88xhl" event={"ID":"8902c3bd-bf64-48e1-b113-d42ed8456348","Type":"ContainerDied","Data":"dd6da7b9b0c6b3c90ae47d10799617ef6acc1dc11708d35296dd689550a2655e"}
Oct 30 20:36:49.464964 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:49.464940    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg" event={"ID":"e30414b5-8768-4eb1-a4ba-395115dba647","Type":"ContainerStarted","Data":"0f6c595e31a6046681a728f6cafdc315281148b143d9e39f0ceb53797757d2e0"}
Oct 30 20:36:49.465046 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:49.465022    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg" event={"ID":"e30414b5-8768-4eb1-a4ba-395115dba647","Type":"ContainerStarted","Data":"49e4dd3614a4ef9bd4039e43c806ceb2cc68c317ed72f5b1a7a459fc0ffb09ed"}
Oct 30 20:36:49.465082 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:49.465073    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg" event={"ID":"e30414b5-8768-4eb1-a4ba-395115dba647","Type":"ContainerStarted","Data":"291f00c24c19a4e2d1ed212999b7cb5c0dd665c8d85106701a75e02d85f82380"}
Oct 30 20:36:49.465132 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:49.465119    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg" event={"ID":"e30414b5-8768-4eb1-a4ba-395115dba647","Type":"ContainerStarted","Data":"391687987bdf76ed4af7ba15822c37f346e0ebf05a728a3d25ea9789dbcb8ca7"}
Oct 30 20:36:49.465224 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:49.465212    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg" event={"ID":"e30414b5-8768-4eb1-a4ba-395115dba647","Type":"ContainerStarted","Data":"b316bfd99ab3a941206794b0c00f8a91c94fbbafc71141fcc2a523c040594694"}
Oct 30 20:36:49.465268 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:49.465257    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg" event={"ID":"e30414b5-8768-4eb1-a4ba-395115dba647","Type":"ContainerStarted","Data":"0b1610aab9105ec52810f73cd53dea1b3727756fa3b944c1523f0ed15ecc9c89"}
Oct 30 20:36:50.004105 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:50.004079    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:50.004263 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:50.004111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:50.004263 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.004216    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:50.004263 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.004257    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret podName:a248c2d8-ba41-4190-9ca7-6cce702dd07a nodeName:}" failed. No retries permitted until 2026-10-30 20:37:06.004245434 +0000 UTC m=+34.139768568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret") pod "global-pull-secret-syncer-vmcq2" (UID: "a248c2d8-ba41-4190-9ca7-6cce702dd07a") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:36:50.004384 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.004218    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:50.004384 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.004331    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs podName:3efcea4c-b6b6-47a5-be1d-c61a83bbe907 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:06.00431951 +0000 UTC m=+34.139842644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs") pod "network-metrics-daemon-gtkw8" (UID: "3efcea4c-b6b6-47a5-be1d-c61a83bbe907") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:36:50.104378 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:50.104342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:50.104516 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.104474    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:36:50.104516 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.104487    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:36:50.104516 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.104496    2576 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksf for pod openshift-network-diagnostics/network-check-target-lfl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:50.104616 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.104538    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf podName:2fc2fd6c-3d77-4446-9402-cf73cdd3497a nodeName:}" failed. No retries permitted until 2026-10-30 20:37:06.104526947 +0000 UTC m=+34.240050093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksf" (UniqueName: "kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf") pod "network-check-target-lfl79" (UID: "2fc2fd6c-3d77-4446-9402-cf73cdd3497a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:36:50.321775 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:50.321700    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:50.321775 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:50.321750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:50.321775 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:50.321761    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:50.322014 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.321881    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:50.322014 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.321909    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:50.322091 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:50.322009    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:50.469789 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:50.469756    2576 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="ca09d1bca6b4ef19b9ebfc2e4f14015c4fcb6da72462c465b8650669ae8907e1" exitCode=0
Oct 30 20:36:50.470215 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:50.469806    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88xhl" event={"ID":"8902c3bd-bf64-48e1-b113-d42ed8456348","Type":"ContainerDied","Data":"ca09d1bca6b4ef19b9ebfc2e4f14015c4fcb6da72462c465b8650669ae8907e1"}
Oct 30 20:36:51.474852 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:51.474807    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-88xhl" event={"ID":"8902c3bd-bf64-48e1-b113-d42ed8456348","Type":"ContainerStarted","Data":"9a34412e52a82194bf4c4eafefb08c9953ed4573ec899d357f31ab3a1c570f21"}
Oct 30 20:36:51.477850 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:51.477826    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg" event={"ID":"e30414b5-8768-4eb1-a4ba-395115dba647","Type":"ContainerStarted","Data":"263c63b6106082549e806d3a6a9112f0c074f360d588d7d0f62eef39b1b6dba3"}
Oct 30 20:36:51.497091 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:51.497056    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-88xhl" podStartSLOduration=8.661747195 podStartE2EDuration="19.497045114s" totalImagesPullingTime="10.835297919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:51.497037936 +0000 UTC m=+19.632561091" watchObservedRunningTime="2026-10-30 20:36:51.497045114 +0000 UTC m=+19.632568246"
Oct 30 20:36:52.322372 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:52.322339    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:52.322546 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:52.322417    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:52.322608 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:52.322518    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:52.322689 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:52.322676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:52.322734 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:52.322679    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:52.322774 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:52.322750    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:53.489410 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:53.489369    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg" event={"ID":"e30414b5-8768-4eb1-a4ba-395115dba647","Type":"ContainerStarted","Data":"1854a2b3f369a98bd81710a7542a99c43068e9b7a059df80a4f1bb3db1605b02"}
Oct 30 20:36:53.490073 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:53.489748    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:53.490073 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:53.489844    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:53.504656 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:53.504634    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:53.518288 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:53.518142    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg" podStartSLOduration=7.327006043 podStartE2EDuration="21.5181311s" totalImagesPullingTime="14.191125057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:53.5175901 +0000 UTC m=+21.653113256" watchObservedRunningTime="2026-10-30 20:36:53.5181311 +0000 UTC m=+21.653654255"
Oct 30 20:36:54.321763 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:54.321732    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:54.321964 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:54.321772    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:54.321964 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:54.321741    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:54.321964 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:54.321847    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:54.321964 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:54.321946    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:54.322109 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:54.321999    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:54.492038 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:54.492001    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:54.507152 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:54.507124    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:36:55.047294 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:55.047264    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vmcq2"]
Oct 30 20:36:55.047439 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:55.047422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:55.047534 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:55.047516    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:55.050614 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:55.050588    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lfl79"]
Oct 30 20:36:55.050746 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:55.050703    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:55.050832 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:55.050806    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:55.051477 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:55.051447    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gtkw8"]
Oct 30 20:36:55.051609 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:55.051595    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:55.051751 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:55.051725    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:56.321434 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:56.321400    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:56.322021 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:56.321510    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:56.322021 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:56.321564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:56.322021 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:56.321649    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:57.321541 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:57.321508    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:57.321936 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:57.321606    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:36:58.321220 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:58.321139    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:36:58.321356 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:58.321237    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:36:58.321356 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:58.321274    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:36:58.321356 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:58.321320    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:36:59.322154 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:36:59.322120    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:36:59.322671 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:36:59.322239    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:37:00.321546 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:00.321514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:37:00.321709 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:00.321655    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gtkw8" podUID="3efcea4c-b6b6-47a5-be1d-c61a83bbe907"
Oct 30 20:37:00.321709 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:00.321696    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:00.321781 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:00.321763    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vmcq2" podUID="a248c2d8-ba41-4190-9ca7-6cce702dd07a"
Oct 30 20:37:01.321175 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.321090    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:37:01.321549 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:01.321211    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lfl79" podUID="2fc2fd6c-3d77-4446-9402-cf73cdd3497a"
Oct 30 20:37:01.666891 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.666800    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeReady"
Oct 30 20:37:01.667022 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.666945    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 20:37:01.706319 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.706286    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nbh76"]
Oct 30 20:37:01.709476 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.709459    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nbh76"
Oct 30 20:37:01.717642 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.717617    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sp5fq"]
Oct 30 20:37:01.720566 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.720549    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jr5cw"]
Oct 30 20:37:01.720690 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.720672    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sp5fq"
Oct 30 20:37:01.720987 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.720962    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.721139 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.721124    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.723254 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.723239    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"]
Oct 30 20:37:01.724883 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.723646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jr5cw"
Oct 30 20:37:01.730888 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.730851    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-948b95b59-tdzdl"]
Oct 30 20:37:01.731002 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.730977    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:01.733926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.733907    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"]
Oct 30 20:37:01.734037 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.734020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.737340 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.737315    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rbf5z"]
Oct 30 20:37:01.738431 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.738414    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:01.740463 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.740443    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jr5cw"]
Oct 30 20:37:01.740568 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.740478    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sp5fq"]
Oct 30 20:37:01.740568 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.740491    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nbh76"]
Oct 30 20:37:01.740568 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.740504    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"]
Oct 30 20:37:01.740568 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.740518    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"]
Oct 30 20:37:01.740568 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.740565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.743371 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.743351    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2"]
Oct 30 20:37:01.743453 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.743444    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:01.744197 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.744174    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 20:37:01.744464 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.744448    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5k9jw\""
Oct 30 20:37:01.746240 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.746210    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2"
Oct 30 20:37:01.746387 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.746298    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-948b95b59-tdzdl"]
Oct 30 20:37:01.747300 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.747280    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 20:37:01.748444 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.748427    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"]
Oct 30 20:37:01.751428 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.751340    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns"]
Oct 30 20:37:01.752066 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.752046    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:01.754292 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.754277    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8nltv"]
Oct 30 20:37:01.754399 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.754386    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns"
Oct 30 20:37:01.757281 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.757262    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7dc99f8759-pzknq"]
Oct 30 20:37:01.757418 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.757387    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:01.760184 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.760168    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bfdcc7f98-zlptz"]
Oct 30 20:37:01.760339 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.760325    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.763169 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.763151    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.770110 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.769612    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"]
Oct 30 20:37:01.772507 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.772491    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nfvzn"]
Oct 30 20:37:01.772658 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.772621    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:01.776969 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.776947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.778505 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.778193    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x9v7g"]
Oct 30 20:37:01.788815 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.788785    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"]
Oct 30 20:37:01.788971 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.788828    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rbf5z"]
Oct 30 20:37:01.788971 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.788849    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2"]
Oct 30 20:37:01.788971 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.788947    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8nltv"]
Oct 30 20:37:01.789374 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.789357    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:01.790954 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.789618    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"]
Oct 30 20:37:01.790954 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.790631    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"]
Oct 30 20:37:01.794016 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.791692    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9v7g"]
Oct 30 20:37:01.794016 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.793534    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7dc99f8759-pzknq"]
Oct 30 20:37:01.794016 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.793567    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns"]
Oct 30 20:37:01.794016 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.793830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwsm\" (UniqueName: \"kubernetes.io/projected/7fc47c20-4a62-4167-9334-4d7b1cadb5b5-kube-api-access-dhwsm\") pod \"kas-connection-checker-685895c7c9-sp5fq\" (UID: \"7fc47c20-4a62-4167-9334-4d7b1cadb5b5\") " pod="kube-system/kas-connection-checker-685895c7c9-sp5fq"
Oct 30 20:37:01.794016 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.793909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkh5r\" (UniqueName: \"kubernetes.io/projected/a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a-kube-api-access-pkh5r\") pod \"kas-connection-checker-685895c7c9-nbh76\" (UID: \"a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a\") " pod="kube-system/kas-connection-checker-685895c7c9-nbh76"
Oct 30 20:37:01.796155 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.796133    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"]
Oct 30 20:37:01.810077 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.810048    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nfvzn"]
Oct 30 20:37:01.812489 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.812360    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bfdcc7f98-zlptz"]
Oct 30 20:37:01.820670 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.820650    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.820840 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.820822    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 20:37:01.820840 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.820833    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.821011 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.820887    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.821011 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.820931    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.821011 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.820937    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.821011 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.820961    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.821011 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821011    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 20:37:01.821232 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821027    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.821232 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821066    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.821232 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821093    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 20:37:01.821232 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821128    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.821232 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821130    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.821232 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821164    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 20:37:01.821232 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821207    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 20:37:01.821232 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821211    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.821232 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821233    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.821476 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821347    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.821583 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821565    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.821661 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821575    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.821661 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821646    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 20:37:01.821738 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821676    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 20:37:01.821940 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.821926    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:37:01.822412 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.822390    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.822487 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.822374    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.822487 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.822426    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:37:01.822487 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.822445    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.822487 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.822428    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:01.825619 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.825598    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 20:37:01.833285 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.833265    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 20:37:01.833770 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.833748    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 20:37:01.833840 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.833827    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 20:37:01.843350 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.843330    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 20:37:01.843414 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.843402    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 20:37:01.843914 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.843892    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 20:37:01.843995 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.843921    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 20:37:01.843995 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.843944    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-srgh4\""
Oct 30 20:37:01.843995 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.843954    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-69hq7\""
Oct 30 20:37:01.843995 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.843989    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 20:37:01.844177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844010    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2zkwq\""
Oct 30 20:37:01.844177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.843938    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 20:37:01.844177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844082    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cg4v8\""
Oct 30 20:37:01.844177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844146    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 20:37:01.844177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844161    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:37:01.844177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844160    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-278b6\""
Oct 30 20:37:01.844432 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844197    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 20:37:01.844432 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844232    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-87d8c\""
Oct 30 20:37:01.844432 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844253    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-btxn9\""
Oct 30 20:37:01.844432 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844267    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 20:37:01.844432 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844288    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:37:01.844432 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844351    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6fg9b\""
Oct 30 20:37:01.844915 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844896    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 20:37:01.845009 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844921    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-sbwq8\""
Oct 30 20:37:01.845009 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844926    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sdwp2\""
Oct 30 20:37:01.845009 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844965    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rsx8b\""
Oct 30 20:37:01.845009 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.844968    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 20:37:01.845009 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.845006    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bpj8f\""
Oct 30 20:37:01.894409 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-trusted-ca\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.894605 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.894605 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-snapshots\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.894605 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/395a48bf-cfce-443e-be13-b2939a9b9b8a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:01.894605 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6140cde0-b52b-48e1-a1d6-bcac35086922-config-volume\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.894605 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894600    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/47697b9d-ed21-4e3b-93b0-320138d4037d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:01.894840 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-bound-sa-token\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.894840 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthfq\" (UniqueName: \"kubernetes.io/projected/b4376a74-205b-45a1-b005-c59fae4f4201-kube-api-access-wthfq\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:01.894840 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-trusted-ca\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:01.894840 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7b7j\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-kube-api-access-v7b7j\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.894840 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-certificates\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.895104 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkh5r\" (UniqueName: \"kubernetes.io/projected/a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a-kube-api-access-pkh5r\") pod \"kas-connection-checker-685895c7c9-nbh76\" (UID: \"a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a\") " pod="kube-system/kas-connection-checker-685895c7c9-nbh76"
Oct 30 20:37:01.895104 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/395a48bf-cfce-443e-be13-b2939a9b9b8a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:01.895104 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:01.895104 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.894982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvtbb\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-kube-api-access-jvtbb\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.895104 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plcbb\" (UniqueName: \"kubernetes.io/projected/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-kube-api-access-plcbb\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.895104 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-tmp\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.895104 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4376a74-205b-45a1-b005-c59fae4f4201-serving-cert\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khjfm\" (UniqueName: \"kubernetes.io/projected/395a48bf-cfce-443e-be13-b2939a9b9b8a-kube-api-access-khjfm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-stats-auth\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8wtn\" (UniqueName: \"kubernetes.io/projected/a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf-kube-api-access-c8wtn\") pod \"network-check-source-6f78d59f99-fhbw2\" (UID: \"a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895271    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scfnr\" (UniqueName: \"kubernetes.io/projected/93f061fe-5c27-419b-8677-9734d5dce379-kube-api-access-scfnr\") pod \"ingress-canary-x9v7g\" (UID: \"93f061fe-5c27-419b-8677-9734d5dce379\") " pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwsm\" (UniqueName: \"kubernetes.io/projected/7fc47c20-4a62-4167-9334-4d7b1cadb5b5-kube-api-access-dhwsm\") pod \"kas-connection-checker-685895c7c9-sp5fq\" (UID: \"7fc47c20-4a62-4167-9334-4d7b1cadb5b5\") " pod="kube-system/kas-connection-checker-685895c7c9-sp5fq"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2m8\" (UniqueName: \"kubernetes.io/projected/8a707e7f-24ce-4322-b5ad-ffaaa21dc23c-kube-api-access-zr2m8\") pod \"kas-connection-checker-685895c7c9-jr5cw\" (UID: \"8a707e7f-24ce-4322-b5ad-ffaaa21dc23c\") " pod="kube-system/kas-connection-checker-685895c7c9-jr5cw"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-config\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:01.895408 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmcl5\" (UniqueName: \"kubernetes.io/projected/6140cde0-b52b-48e1-a1d6-bcac35086922-kube-api-access-wmcl5\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/204ba748-a514-4772-9fba-dfafd3eddcf7-ca-trust-extracted\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895471    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-installation-pull-secrets\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-bound-sa-token\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dd510fb-f40e-401c-845f-23b9d750dc68-ca-trust-extracted\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-default-certificate\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4376a74-205b-45a1-b005-c59fae4f4201-config\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-service-ca-bundle\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895713    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msb4h\" (UniqueName: \"kubernetes.io/projected/b3e2fc05-6adb-4c94-a124-aca4483a4e13-kube-api-access-msb4h\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-certificates\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6140cde0-b52b-48e1-a1d6-bcac35086922-tmp-dir\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtw88\" (UniqueName: \"kubernetes.io/projected/47697b9d-ed21-4e3b-93b0-320138d4037d-kube-api-access-dtw88\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:01.895926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895839    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-installation-pull-secrets\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93f061fe-5c27-419b-8677-9734d5dce379-cert\") pod \"ingress-canary-x9v7g\" (UID: \"93f061fe-5c27-419b-8677-9734d5dce379\") " pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-image-registry-private-configuration\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895933    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebc0553b-387f-438a-ab73-a053b37abcdd-serving-cert\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-image-registry-private-configuration\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.895997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3e2fc05-6adb-4c94-a124-aca4483a4e13-serving-cert\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.896021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.896059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjmlj\" (UniqueName: \"kubernetes.io/projected/f7aac07e-1e6f-4f62-897c-0061aa870749-kube-api-access-zjmlj\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.896087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvvh\" (UniqueName: \"kubernetes.io/projected/ebc0553b-387f-438a-ab73-a053b37abcdd-kube-api-access-2jvvh\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.896138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-trusted-ca\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.896215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b94nz\" (UniqueName: \"kubernetes.io/projected/0d7898c1-9d6c-4382-baa2-e6baae34941a-kube-api-access-b94nz\") pod \"volume-data-source-validator-58c9f5f46d-klqns\" (UID: \"0d7898c1-9d6c-4382-baa2-e6baae34941a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.896246    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/734323fa-6946-426b-9e06-396cb4e7b48f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:01.896502 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.896287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.905532 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.905417    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkh5r\" (UniqueName: \"kubernetes.io/projected/a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a-kube-api-access-pkh5r\") pod \"kas-connection-checker-685895c7c9-nbh76\" (UID: \"a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a\") " pod="kube-system/kas-connection-checker-685895c7c9-nbh76"
Oct 30 20:37:01.905655 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.905420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwsm\" (UniqueName: \"kubernetes.io/projected/7fc47c20-4a62-4167-9334-4d7b1cadb5b5-kube-api-access-dhwsm\") pod \"kas-connection-checker-685895c7c9-sp5fq\" (UID: \"7fc47c20-4a62-4167-9334-4d7b1cadb5b5\") " pod="kube-system/kas-connection-checker-685895c7c9-sp5fq"
Oct 30 20:37:01.944055 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.943970    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:37:01.944217 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.944111    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:37:01.944653 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.944637    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:37:01.996672 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.996638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4376a74-205b-45a1-b005-c59fae4f4201-config\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:01.996672 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.996670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-service-ca-bundle\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.996935 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.996687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msb4h\" (UniqueName: \"kubernetes.io/projected/b3e2fc05-6adb-4c94-a124-aca4483a4e13-kube-api-access-msb4h\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.996935 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.996897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-certificates\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.997016 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.996935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6140cde0-b52b-48e1-a1d6-bcac35086922-tmp-dir\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.997016 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.996966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtw88\" (UniqueName: \"kubernetes.io/projected/47697b9d-ed21-4e3b-93b0-320138d4037d-kube-api-access-dtw88\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:01.997016 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.996992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-installation-pull-secrets\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.997016 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93f061fe-5c27-419b-8677-9734d5dce379-cert\") pod \"ingress-canary-x9v7g\" (UID: \"93f061fe-5c27-419b-8677-9734d5dce379\") " pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:01.997190 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-image-registry-private-configuration\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.997190 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebc0553b-387f-438a-ab73-a053b37abcdd-serving-cert\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:01.997190 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-image-registry-private-configuration\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.997190 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3e2fc05-6adb-4c94-a124-aca4483a4e13-serving-cert\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.997190 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.997190 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjmlj\" (UniqueName: \"kubernetes.io/projected/f7aac07e-1e6f-4f62-897c-0061aa870749-kube-api-access-zjmlj\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:01.997190 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvvh\" (UniqueName: \"kubernetes.io/projected/ebc0553b-387f-438a-ab73-a053b37abcdd-kube-api-access-2jvvh\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:01.997406 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-trusted-ca\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.997406 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94nz\" (UniqueName: \"kubernetes.io/projected/0d7898c1-9d6c-4382-baa2-e6baae34941a-kube-api-access-b94nz\") pod \"volume-data-source-validator-58c9f5f46d-klqns\" (UID: \"0d7898c1-9d6c-4382-baa2-e6baae34941a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns"
Oct 30 20:37:01.997406 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/734323fa-6946-426b-9e06-396cb4e7b48f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:01.997406 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.997406 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997310    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6140cde0-b52b-48e1-a1d6-bcac35086922-tmp-dir\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.997406 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-trusted-ca\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.997406 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.997406 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-snapshots\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997435    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/395a48bf-cfce-443e-be13-b2939a9b9b8a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6140cde0-b52b-48e1-a1d6-bcac35086922-config-volume\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997475    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4376a74-205b-45a1-b005-c59fae4f4201-config\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/47697b9d-ed21-4e3b-93b0-320138d4037d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-bound-sa-token\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthfq\" (UniqueName: \"kubernetes.io/projected/b4376a74-205b-45a1-b005-c59fae4f4201-kube-api-access-wthfq\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:01.997564    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:01.997579    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfdcc7f98-zlptz: secret "image-registry-tls" not found
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997577    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-certificates\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:01.997638    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls podName:204ba748-a514-4772-9fba-dfafd3eddcf7 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:02.497621534 +0000 UTC m=+30.633144683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls") pod "image-registry-5bfdcc7f98-zlptz" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7") : secret "image-registry-tls" not found
Oct 30 20:37:01.997655 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997478    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-service-ca-bundle\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.998177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-trusted-ca\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:01.998177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7b7j\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-kube-api-access-v7b7j\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.998177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-certificates\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.998177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.997986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/395a48bf-cfce-443e-be13-b2939a9b9b8a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:01.998177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:01.998177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtbb\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-kube-api-access-jvtbb\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.998177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plcbb\" (UniqueName: \"kubernetes.io/projected/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-kube-api-access-plcbb\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.998177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-tmp\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.998177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4376a74-205b-45a1-b005-c59fae4f4201-serving-cert\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:01.998177 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khjfm\" (UniqueName: \"kubernetes.io/projected/395a48bf-cfce-443e-be13-b2939a9b9b8a-kube-api-access-khjfm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-stats-auth\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8wtn\" (UniqueName: \"kubernetes.io/projected/a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf-kube-api-access-c8wtn\") pod \"network-check-source-6f78d59f99-fhbw2\" (UID: \"a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfnr\" (UniqueName: \"kubernetes.io/projected/93f061fe-5c27-419b-8677-9734d5dce379-kube-api-access-scfnr\") pod \"ingress-canary-x9v7g\" (UID: \"93f061fe-5c27-419b-8677-9734d5dce379\") " pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2m8\" (UniqueName: \"kubernetes.io/projected/8a707e7f-24ce-4322-b5ad-ffaaa21dc23c-kube-api-access-zr2m8\") pod \"kas-connection-checker-685895c7c9-jr5cw\" (UID: \"8a707e7f-24ce-4322-b5ad-ffaaa21dc23c\") " pod="kube-system/kas-connection-checker-685895c7c9-jr5cw"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998363    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/47697b9d-ed21-4e3b-93b0-320138d4037d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-config\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998400    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmcl5\" (UniqueName: \"kubernetes.io/projected/6140cde0-b52b-48e1-a1d6-bcac35086922-kube-api-access-wmcl5\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/204ba748-a514-4772-9fba-dfafd3eddcf7-ca-trust-extracted\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-installation-pull-secrets\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-bound-sa-token\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:01.998640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:01.999385 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dd510fb-f40e-401c-845f-23b9d750dc68-ca-trust-extracted\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:01.999385 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-default-certificate\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:01.999385 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-trusted-ca\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:01.999385 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.998901    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-snapshots\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.999385 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.999224    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6140cde0-b52b-48e1-a1d6-bcac35086922-config-volume\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:01.999619 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:01.999391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-tmp\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:01.999619 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:01.999516    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:37:01.999619 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:01.999562    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls podName:6140cde0-b52b-48e1-a1d6-bcac35086922 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:02.499549405 +0000 UTC m=+30.635072552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls") pod "dns-default-nfvzn" (UID: "6140cde0-b52b-48e1-a1d6-bcac35086922") : secret "dns-default-metrics-tls" not found
Oct 30 20:37:01.999951 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:01.999923    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:37:02.001753 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.000584    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-certificates\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:02.001753 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.000627    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/395a48bf-cfce-443e-be13-b2939a9b9b8a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:02.001753 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.000940    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-image-registry-private-configuration\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.001376    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-config\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.001441    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/204ba748-a514-4772-9fba-dfafd3eddcf7-ca-trust-extracted\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.001790    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-trusted-ca\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.000723    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.001539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/395a48bf-cfce-443e-be13-b2939a9b9b8a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.001422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-image-registry-private-configuration\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.000774    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.001549    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:02.50153204 +0000 UTC m=+30.637055192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : configmap references non-existent config key: service-ca.crt
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.001929    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert podName:734323fa-6946-426b-9e06-396cb4e7b48f nodeName:}" failed. No retries permitted until 2026-10-30 20:37:02.501910159 +0000 UTC m=+30.637433299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wt7nr" (UID: "734323fa-6946-426b-9e06-396cb4e7b48f") : secret "networking-console-plugin-cert" not found
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.001961    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:02.501949065 +0000 UTC m=+30.637472199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : secret "router-metrics-certs-default" not found
Oct 30 20:37:02.002015 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.001982    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls podName:47697b9d-ed21-4e3b-93b0-320138d4037d nodeName:}" failed. No retries permitted until 2026-10-30 20:37:02.501972733 +0000 UTC m=+30.637495881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cd75b" (UID: "47697b9d-ed21-4e3b-93b0-320138d4037d") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:37:02.002540 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.002024    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93f061fe-5c27-419b-8677-9734d5dce379-cert\") pod \"ingress-canary-x9v7g\" (UID: \"93f061fe-5c27-419b-8677-9734d5dce379\") " pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:02.002540 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.002119    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:02.002540 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.002409    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:02.002540 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.002414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dd510fb-f40e-401c-845f-23b9d750dc68-ca-trust-extracted\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:02.002540 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.002426    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-948b95b59-tdzdl: secret "image-registry-tls" not found
Oct 30 20:37:02.002540 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.002480    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls podName:4dd510fb-f40e-401c-845f-23b9d750dc68 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:02.502465521 +0000 UTC m=+30.637988669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls") pod "image-registry-948b95b59-tdzdl" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68") : secret "image-registry-tls" not found
Oct 30 20:37:02.002540 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.002535    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/734323fa-6946-426b-9e06-396cb4e7b48f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:02.002896 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.002555    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:37:02.002896 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.002594    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls podName:f7aac07e-1e6f-4f62-897c-0061aa870749 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:02.502581395 +0000 UTC m=+30.638104573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-crfsg" (UID: "f7aac07e-1e6f-4f62-897c-0061aa870749") : secret "samples-operator-tls" not found
Oct 30 20:37:02.002896 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.002631    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:02.003058 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.003001    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-trusted-ca\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:02.003269 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.003250    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3e2fc05-6adb-4c94-a124-aca4483a4e13-serving-cert\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:02.004363 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.004334    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-installation-pull-secrets\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:02.004508 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.004487    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-stats-auth\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:02.004589 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.004568    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebc0553b-387f-438a-ab73-a053b37abcdd-serving-cert\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:02.004699 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.004683    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4376a74-205b-45a1-b005-c59fae4f4201-serving-cert\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:02.004800 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.004774    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-installation-pull-secrets\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:02.005782 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.005765    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-default-certificate\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:02.006554 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.006536    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msb4h\" (UniqueName: \"kubernetes.io/projected/b3e2fc05-6adb-4c94-a124-aca4483a4e13-kube-api-access-msb4h\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:02.006619 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.006542    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtw88\" (UniqueName: \"kubernetes.io/projected/47697b9d-ed21-4e3b-93b0-320138d4037d-kube-api-access-dtw88\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:02.011318 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.011000    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfnr\" (UniqueName: \"kubernetes.io/projected/93f061fe-5c27-419b-8677-9734d5dce379-kube-api-access-scfnr\") pod \"ingress-canary-x9v7g\" (UID: \"93f061fe-5c27-419b-8677-9734d5dce379\") " pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:02.011318 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.011131    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2m8\" (UniqueName: \"kubernetes.io/projected/8a707e7f-24ce-4322-b5ad-ffaaa21dc23c-kube-api-access-zr2m8\") pod \"kas-connection-checker-685895c7c9-jr5cw\" (UID: \"8a707e7f-24ce-4322-b5ad-ffaaa21dc23c\") " pod="kube-system/kas-connection-checker-685895c7c9-jr5cw"
Oct 30 20:37:02.011538 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.011513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthfq\" (UniqueName: \"kubernetes.io/projected/b4376a74-205b-45a1-b005-c59fae4f4201-kube-api-access-wthfq\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:02.011592 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.011514    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plcbb\" (UniqueName: \"kubernetes.io/projected/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-kube-api-access-plcbb\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:02.011642 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.011593    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-bound-sa-token\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:02.012452 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.012429    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7b7j\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-kube-api-access-v7b7j\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:02.013928 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.013853    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8wtn\" (UniqueName: \"kubernetes.io/projected/a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf-kube-api-access-c8wtn\") pod \"network-check-source-6f78d59f99-fhbw2\" (UID: \"a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2"
Oct 30 20:37:02.014220 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.014023    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvtbb\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-kube-api-access-jvtbb\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:02.014220 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.014208    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94nz\" (UniqueName: \"kubernetes.io/projected/0d7898c1-9d6c-4382-baa2-e6baae34941a-kube-api-access-b94nz\") pod \"volume-data-source-validator-58c9f5f46d-klqns\" (UID: \"0d7898c1-9d6c-4382-baa2-e6baae34941a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns"
Oct 30 20:37:02.015112 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.015009    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmcl5\" (UniqueName: \"kubernetes.io/projected/6140cde0-b52b-48e1-a1d6-bcac35086922-kube-api-access-wmcl5\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:02.015112 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.015060    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjmlj\" (UniqueName: \"kubernetes.io/projected/f7aac07e-1e6f-4f62-897c-0061aa870749-kube-api-access-zjmlj\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:02.015989 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.015968    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvvh\" (UniqueName: \"kubernetes.io/projected/ebc0553b-387f-438a-ab73-a053b37abcdd-kube-api-access-2jvvh\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:02.016216 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.016193    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khjfm\" (UniqueName: \"kubernetes.io/projected/395a48bf-cfce-443e-be13-b2939a9b9b8a-kube-api-access-khjfm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:02.016769 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.016755    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-bound-sa-token\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:02.022143 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.022127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nbh76"
Oct 30 20:37:02.033435 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.033411    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sp5fq"
Oct 30 20:37:02.042016 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.041996    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jr5cw"
Oct 30 20:37:02.051247 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.051220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:02.066641 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.066381    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:02.077423 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.077386    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:02.098013 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.097816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2"
Oct 30 20:37:02.117563 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.117217    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns"
Oct 30 20:37:02.123161 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.122059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:02.155824 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.154924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:02.158334 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.158028    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:02.296093 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.296054    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:02.322990 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.322964    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:02.323763 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.323131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:37:02.328419 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.328382    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sp5fq"]
Oct 30 20:37:02.332962 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.332939    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nbh76"]
Oct 30 20:37:02.337237 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.336142    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jr5cw"]
Oct 30 20:37:02.337237 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.333546    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zljsp\""
Oct 30 20:37:02.337237 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.333405    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jrl7k\""
Oct 30 20:37:02.337237 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.333649    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:37:02.337237 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.333768    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:37:02.366635 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.366604    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"]
Oct 30 20:37:02.367967 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:37:02.367939    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4376a74_205b_45a1_b005_c59fae4f4201.slice/crio-e454def1ef9c855b78b8c4a7113721c71551983116c56705ddb8cd79a617fd43 WatchSource:0}: Error finding container e454def1ef9c855b78b8c4a7113721c71551983116c56705ddb8cd79a617fd43: Status 404 returned error can't find the container with id e454def1ef9c855b78b8c4a7113721c71551983116c56705ddb8cd79a617fd43
Oct 30 20:37:02.376321 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.376274    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"]
Oct 30 20:37:02.399032 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.398959    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rbf5z"]
Oct 30 20:37:02.438276 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.438247    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2"]
Oct 30 20:37:02.442568 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:37:02.442538    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96bf55a_0e4c_4952_b7fc_4f20c3f84fdf.slice/crio-e17ab44add68a3afb1672716b073e1a15d617e833b45b102dde94aa0d60f74e7 WatchSource:0}: Error finding container e17ab44add68a3afb1672716b073e1a15d617e833b45b102dde94aa0d60f74e7: Status 404 returned error can't find the container with id e17ab44add68a3afb1672716b073e1a15d617e833b45b102dde94aa0d60f74e7
Oct 30 20:37:02.456614 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.456475    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns"]
Oct 30 20:37:02.460036 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.460017    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8nltv"]
Oct 30 20:37:02.471085 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.471062    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9v7g"]
Oct 30 20:37:02.481834 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:37:02.481809    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f061fe_5c27_419b_8677_9734d5dce379.slice/crio-b3a0c9d437acf8b3dae52da884f0c764c3f8f8b01bf4947dd9a149c635bc1584 WatchSource:0}: Error finding container b3a0c9d437acf8b3dae52da884f0c764c3f8f8b01bf4947dd9a149c635bc1584: Status 404 returned error can't find the container with id b3a0c9d437acf8b3dae52da884f0c764c3f8f8b01bf4947dd9a149c635bc1584
Oct 30 20:37:02.507927 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.507898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:02.508021 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.507968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:02.508021 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.508010    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:02.508138 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.508047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:02.508138 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.508084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:02.508138 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508122    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:37:02.508283 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508147    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:37:02.508283 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.508164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:02.508283 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508185    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls podName:47697b9d-ed21-4e3b-93b0-320138d4037d nodeName:}" failed. No retries permitted until 2026-10-30 20:37:03.508165427 +0000 UTC m=+31.643688575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cd75b" (UID: "47697b9d-ed21-4e3b-93b0-320138d4037d") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:37:02.508283 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508210    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:37:02.508283 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508147    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:37:02.508283 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508261    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert podName:734323fa-6946-426b-9e06-396cb4e7b48f nodeName:}" failed. No retries permitted until 2026-10-30 20:37:03.508244624 +0000 UTC m=+31.643767758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wt7nr" (UID: "734323fa-6946-426b-9e06-396cb4e7b48f") : secret "networking-console-plugin-cert" not found
Oct 30 20:37:02.508283 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508249    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:37:02.508283 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508281    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls podName:f7aac07e-1e6f-4f62-897c-0061aa870749 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:03.508271866 +0000 UTC m=+31.643795006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-crfsg" (UID: "f7aac07e-1e6f-4f62-897c-0061aa870749") : secret "samples-operator-tls" not found
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.508305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508318    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:03.508301718 +0000 UTC m=+31.643824862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : secret "router-metrics-certs-default" not found
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508348    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls podName:6140cde0-b52b-48e1-a1d6-bcac35086922 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:03.508335176 +0000 UTC m=+31.643858312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls") pod "dns-default-nfvzn" (UID: "6140cde0-b52b-48e1-a1d6-bcac35086922") : secret "dns-default-metrics-tls" not found
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508372    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508382    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:03.508371773 +0000 UTC m=+31.643894912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : configmap references non-existent config key: service-ca.crt
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508384    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-948b95b59-tdzdl: secret "image-registry-tls" not found
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.508404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508423    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls podName:4dd510fb-f40e-401c-845f-23b9d750dc68 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:03.508412581 +0000 UTC m=+31.643935717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls") pod "image-registry-948b95b59-tdzdl" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68") : secret "image-registry-tls" not found
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508480    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508491    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfdcc7f98-zlptz: secret "image-registry-tls" not found
Oct 30 20:37:02.508640 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:02.508527    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls podName:204ba748-a514-4772-9fba-dfafd3eddcf7 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:03.508517112 +0000 UTC m=+31.644040296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls") pod "image-registry-5bfdcc7f98-zlptz" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7") : secret "image-registry-tls" not found
Oct 30 20:37:02.510983 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.510957    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n" event={"ID":"395a48bf-cfce-443e-be13-b2939a9b9b8a","Type":"ContainerStarted","Data":"591c79834fe538885aa02a531b5fa1849744a49dda1c869cca457167cd429429"}
Oct 30 20:37:02.512530 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.512507    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sp5fq" event={"ID":"7fc47c20-4a62-4167-9334-4d7b1cadb5b5","Type":"ContainerStarted","Data":"07fed86e43c76d81b6022feecf6da951b32878b9bc7e7ba5b1a89ab32dc8e6ab"}
Oct 30 20:37:02.512607 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.512541    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sp5fq" event={"ID":"7fc47c20-4a62-4167-9334-4d7b1cadb5b5","Type":"ContainerStarted","Data":"4eccb371c44a504ecceba60e71d99b241c6719656d1e3153e55d3e4c58cdc01f"}
Oct 30 20:37:02.513985 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.513965    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rbf5z" event={"ID":"b3e2fc05-6adb-4c94-a124-aca4483a4e13","Type":"ContainerStarted","Data":"1d4395f4806e92dfb0cbfee2babccbd31159975836f2db0df3a1548716eb312f"}
Oct 30 20:37:02.515109 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.515081    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9v7g" event={"ID":"93f061fe-5c27-419b-8677-9734d5dce379","Type":"ContainerStarted","Data":"b3a0c9d437acf8b3dae52da884f0c764c3f8f8b01bf4947dd9a149c635bc1584"}
Oct 30 20:37:02.517244 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.517205    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" event={"ID":"ebc0553b-387f-438a-ab73-a053b37abcdd","Type":"ContainerStarted","Data":"6b2528cedc5eec4e212563e57e0ff38931265057f3bf59f9d1d47d51a42f2284"}
Oct 30 20:37:02.519191 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.519172    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jr5cw" event={"ID":"8a707e7f-24ce-4322-b5ad-ffaaa21dc23c","Type":"ContainerStarted","Data":"43aa7dbc9aa4e992207ee9f8f38528a0a6677449695534d8cff80e7d8750ca4a"}
Oct 30 20:37:02.519285 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.519212    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jr5cw" event={"ID":"8a707e7f-24ce-4322-b5ad-ffaaa21dc23c","Type":"ContainerStarted","Data":"6ed8f5dbe5f679f34e4c7e968461203bd8f366a76aebfff9e050a3c135e76bc1"}
Oct 30 20:37:02.520233 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.520215    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2" event={"ID":"a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf","Type":"ContainerStarted","Data":"e17ab44add68a3afb1672716b073e1a15d617e833b45b102dde94aa0d60f74e7"}
Oct 30 20:37:02.521173 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.521152    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd" event={"ID":"b4376a74-205b-45a1-b005-c59fae4f4201","Type":"ContainerStarted","Data":"e454def1ef9c855b78b8c4a7113721c71551983116c56705ddb8cd79a617fd43"}
Oct 30 20:37:02.522452 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.522433    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nbh76" event={"ID":"a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a","Type":"ContainerStarted","Data":"885d0eb753c174f5bdaadee8b97ec31d9e825f5f6ff68457ec5b7274e3164e13"}
Oct 30 20:37:02.522546 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.522464    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nbh76" event={"ID":"a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a","Type":"ContainerStarted","Data":"84c4b87f449036eaa626d4e1149ae5e7c367127b152e827259d9f91129c5a168"}
Oct 30 20:37:02.523536 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.523493    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns" event={"ID":"0d7898c1-9d6c-4382-baa2-e6baae34941a","Type":"ContainerStarted","Data":"b53bf2d99bd9bb10291cc1130c05774b10c78c543f5a74db35199f5b662b06d5"}
Oct 30 20:37:02.545379 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.545323    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-sp5fq" podStartSLOduration=395.545310811 podStartE2EDuration="6m35.545310811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:02.528582648 +0000 UTC m=+30.664105804" watchObservedRunningTime="2026-10-30 20:37:02.545310811 +0000 UTC m=+30.680834000"
Oct 30 20:37:02.545788 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.545747    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nbh76" podStartSLOduration=395.545739428 podStartE2EDuration="6m35.545739428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:02.544490052 +0000 UTC m=+30.680013207" watchObservedRunningTime="2026-10-30 20:37:02.545739428 +0000 UTC m=+30.681262583"
Oct 30 20:37:02.564657 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:02.564581    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jr5cw" podStartSLOduration=395.564551109 podStartE2EDuration="6m35.564551109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:02.562323057 +0000 UTC m=+30.697846213" watchObservedRunningTime="2026-10-30 20:37:02.564551109 +0000 UTC m=+30.700074265"
Oct 30 20:37:03.321553 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.321518    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:37:03.326720 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.326696    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4pvnz\""
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.529449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.529526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.529566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.529608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.529695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.529765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.529827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.529921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.530082    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.530138    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls podName:47697b9d-ed21-4e3b-93b0-320138d4037d nodeName:}" failed. No retries permitted until 2026-10-30 20:37:05.530124345 +0000 UTC m=+33.665647491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cd75b" (UID: "47697b9d-ed21-4e3b-93b0-320138d4037d") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.530506    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.530551    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls podName:f7aac07e-1e6f-4f62-897c-0061aa870749 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:05.530535446 +0000 UTC m=+33.666058580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-crfsg" (UID: "f7aac07e-1e6f-4f62-897c-0061aa870749") : secret "samples-operator-tls" not found
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.530598    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:37:03.530683 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.530623    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls podName:6140cde0-b52b-48e1-a1d6-bcac35086922 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:05.530615631 +0000 UTC m=+33.666138763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls") pod "dns-default-nfvzn" (UID: "6140cde0-b52b-48e1-a1d6-bcac35086922") : secret "dns-default-metrics-tls" not found
Oct 30 20:37:03.535075 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535050    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:37:03.535212 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535120    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert podName:734323fa-6946-426b-9e06-396cb4e7b48f nodeName:}" failed. No retries permitted until 2026-10-30 20:37:05.535095997 +0000 UTC m=+33.670619148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wt7nr" (UID: "734323fa-6946-426b-9e06-396cb4e7b48f") : secret "networking-console-plugin-cert" not found
Oct 30 20:37:03.535212 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535129    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:03.535212 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535143    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-948b95b59-tdzdl: secret "image-registry-tls" not found
Oct 30 20:37:03.535212 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535190    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls podName:4dd510fb-f40e-401c-845f-23b9d750dc68 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:05.53517716 +0000 UTC m=+33.670700302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls") pod "image-registry-948b95b59-tdzdl" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68") : secret "image-registry-tls" not found
Oct 30 20:37:03.535423 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535282    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:05.535270528 +0000 UTC m=+33.670793880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : configmap references non-existent config key: service-ca.crt
Oct 30 20:37:03.535423 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535378    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:37:03.535521 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535432    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:03.535521 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535448    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfdcc7f98-zlptz: secret "image-registry-tls" not found
Oct 30 20:37:03.535521 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535451    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:05.535433911 +0000 UTC m=+33.670957047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : secret "router-metrics-certs-default" not found
Oct 30 20:37:03.535521 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:03.535497    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls podName:204ba748-a514-4772-9fba-dfafd3eddcf7 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:05.535483377 +0000 UTC m=+33.671006525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls") pod "image-registry-5bfdcc7f98-zlptz" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7") : secret "image-registry-tls" not found
Oct 30 20:37:03.749135 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.749039    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:03.823285 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:03.823224    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:05.553428 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.552299    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns" event={"ID":"0d7898c1-9d6c-4382-baa2-e6baae34941a","Type":"ContainerStarted","Data":"a0c3840c8343453c31e3cfdba62b605452be50637dc44e7f677ea8399748fe28"}
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.554270    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.554358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.554433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554473    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:09.554451716 +0000 UTC m=+37.689974855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : configmap references non-existent config key: service-ca.crt
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.554506    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554516    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554532    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554545    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-948b95b59-tdzdl: secret "image-registry-tls" not found
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554577    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:09.554559957 +0000 UTC m=+37.690083110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : secret "router-metrics-certs-default" not found
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554602    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls podName:4dd510fb-f40e-401c-845f-23b9d750dc68 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:09.554591966 +0000 UTC m=+37.690115110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls") pod "image-registry-948b95b59-tdzdl" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68") : secret "image-registry-tls" not found
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554633    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554646    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfdcc7f98-zlptz: secret "image-registry-tls" not found
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.554653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554685    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls podName:204ba748-a514-4772-9fba-dfafd3eddcf7 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:09.55467311 +0000 UTC m=+37.690196245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls") pod "image-registry-5bfdcc7f98-zlptz" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7") : secret "image-registry-tls" not found
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554726    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:37:05.555295 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.554733    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:05.556154 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554762    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls podName:47697b9d-ed21-4e3b-93b0-320138d4037d nodeName:}" failed. No retries permitted until 2026-10-30 20:37:09.554752665 +0000 UTC m=+37.690275804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cd75b" (UID: "47697b9d-ed21-4e3b-93b0-320138d4037d") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:37:05.556154 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.554790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:05.556154 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.554817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:05.556154 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554938    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:37:05.556154 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554942    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:37:05.556154 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554978    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert podName:734323fa-6946-426b-9e06-396cb4e7b48f nodeName:}" failed. No retries permitted until 2026-10-30 20:37:09.554966801 +0000 UTC m=+37.690489936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wt7nr" (UID: "734323fa-6946-426b-9e06-396cb4e7b48f") : secret "networking-console-plugin-cert" not found
Oct 30 20:37:05.556154 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.554993    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls podName:f7aac07e-1e6f-4f62-897c-0061aa870749 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:09.554984793 +0000 UTC m=+37.690507947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-crfsg" (UID: "f7aac07e-1e6f-4f62-897c-0061aa870749") : secret "samples-operator-tls" not found
Oct 30 20:37:05.556154 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.555029    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:37:05.556154 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:05.555062    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls podName:6140cde0-b52b-48e1-a1d6-bcac35086922 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:09.555050974 +0000 UTC m=+37.690574130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls") pod "dns-default-nfvzn" (UID: "6140cde0-b52b-48e1-a1d6-bcac35086922") : secret "dns-default-metrics-tls" not found
Oct 30 20:37:05.570170 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.570132    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns" podStartSLOduration=343.487331801 podStartE2EDuration="5m45.570121509s" totalImagesPullingTime="2.082789708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:05.568578887 +0000 UTC m=+33.704102044" watchObservedRunningTime="2026-10-30 20:37:05.570121509 +0000 UTC m=+33.705644675"
Oct 30 20:37:05.693596 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.693525    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:05.763846 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:05.763752    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:06.061214 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:06.061169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:37:06.061411 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:06.061229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:06.062516 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:06.062167    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:37:06.062516 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:06.062237    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs podName:3efcea4c-b6b6-47a5-be1d-c61a83bbe907 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:38.062218126 +0000 UTC m=+66.197741283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs") pod "network-metrics-daemon-gtkw8" (UID: "3efcea4c-b6b6-47a5-be1d-c61a83bbe907") : secret "metrics-daemon-secret" not found
Oct 30 20:37:06.066524 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:06.066488    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:06.163167 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:06.162741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:37:06.165985 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:06.165924    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:37:06.245956 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:06.245914    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:06.340660 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:06.340422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:37:06.720590 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:06.720145    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vmcq2"]
Oct 30 20:37:06.723721 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:06.723264    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lfl79"]
Oct 30 20:37:06.800425 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:37:06.800386    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda248c2d8_ba41_4190_9ca7_6cce702dd07a.slice/crio-6b23ef6f859a17d83bde3e36d35d2ae515a76bfea55f5c2762c57472d70669a2 WatchSource:0}: Error finding container 6b23ef6f859a17d83bde3e36d35d2ae515a76bfea55f5c2762c57472d70669a2: Status 404 returned error can't find the container with id 6b23ef6f859a17d83bde3e36d35d2ae515a76bfea55f5c2762c57472d70669a2
Oct 30 20:37:06.803348 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:37:06.803320    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc2fd6c_3d77_4446_9402_cf73cdd3497a.slice/crio-0fc7e09d5a272e8b6c1a912933f9ed70ff39777b9df34e476cda8442364a703b WatchSource:0}: Error finding container 0fc7e09d5a272e8b6c1a912933f9ed70ff39777b9df34e476cda8442364a703b: Status 404 returned error can't find the container with id 0fc7e09d5a272e8b6c1a912933f9ed70ff39777b9df34e476cda8442364a703b
Oct 30 20:37:07.560619 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.560584    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vmcq2" event={"ID":"a248c2d8-ba41-4190-9ca7-6cce702dd07a","Type":"ContainerStarted","Data":"6b23ef6f859a17d83bde3e36d35d2ae515a76bfea55f5c2762c57472d70669a2"}
Oct 30 20:37:07.562351 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.562317    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n" event={"ID":"395a48bf-cfce-443e-be13-b2939a9b9b8a","Type":"ContainerStarted","Data":"e25786359b81e42d236510c30175f63a6035115b38804dc067d64f993f71ee71"}
Oct 30 20:37:07.563843 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.563779    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lfl79" event={"ID":"2fc2fd6c-3d77-4446-9402-cf73cdd3497a","Type":"ContainerStarted","Data":"0fc7e09d5a272e8b6c1a912933f9ed70ff39777b9df34e476cda8442364a703b"}
Oct 30 20:37:07.566024 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.566004    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/0.log"
Oct 30 20:37:07.566125 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.566046    2576 generic.go:356] "Generic (PLEG): container finished" podID="ebc0553b-387f-438a-ab73-a053b37abcdd" containerID="0f4f50129487ec968f83493c52eee6745af4d4b6cb7c6bbcdc2adbd910c8c9fc" exitCode=255
Oct 30 20:37:07.566125 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.566086    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" event={"ID":"ebc0553b-387f-438a-ab73-a053b37abcdd","Type":"ContainerDied","Data":"0f4f50129487ec968f83493c52eee6745af4d4b6cb7c6bbcdc2adbd910c8c9fc"}
Oct 30 20:37:07.566535 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.566514    2576 scope.go:122] "RemoveContainer" containerID="0f4f50129487ec968f83493c52eee6745af4d4b6cb7c6bbcdc2adbd910c8c9fc"
Oct 30 20:37:07.579144 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.579102    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n" podStartSLOduration=343.078769691 podStartE2EDuration="5m47.579089097s" totalImagesPullingTime="4.500319406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:07.577972959 +0000 UTC m=+35.713496115" watchObservedRunningTime="2026-10-30 20:37:07.579089097 +0000 UTC m=+35.714612269"
Oct 30 20:37:07.646805 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.646756    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:07.653942 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.652303    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf"]
Oct 30 20:37:07.666182 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.666145    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq"]
Oct 30 20:37:07.666340 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.666318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf"
Oct 30 20:37:07.680321 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.678918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfsd\" (UniqueName: \"kubernetes.io/projected/2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a-kube-api-access-wxfsd\") pod \"migrator-66df5d496b-xpdvf\" (UID: \"2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf"
Oct 30 20:37:07.680321 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.679069    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:07.680321 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.680030    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 20:37:07.680321 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.680112    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-49kbz\""
Oct 30 20:37:07.680692 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.680416    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf"]
Oct 30 20:37:07.680692 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.680548    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq"
Oct 30 20:37:07.682363 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.682215    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq"]
Oct 30 20:37:07.745149 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.745117    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:07.780292 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.780266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfsd\" (UniqueName: \"kubernetes.io/projected/2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a-kube-api-access-wxfsd\") pod \"migrator-66df5d496b-xpdvf\" (UID: \"2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf"
Oct 30 20:37:07.780404 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.780387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdc5\" (UniqueName: \"kubernetes.io/projected/72c43d73-596a-48d7-aa34-60c720b92fb6-kube-api-access-khdc5\") pod \"migrator-66df5d496b-vrlcq\" (UID: \"72c43d73-596a-48d7-aa34-60c720b92fb6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq"
Oct 30 20:37:07.795321 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.795270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfsd\" (UniqueName: \"kubernetes.io/projected/2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a-kube-api-access-wxfsd\") pod \"migrator-66df5d496b-xpdvf\" (UID: \"2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf"
Oct 30 20:37:07.849434 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.849366    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:07.881887 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.881808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdc5\" (UniqueName: \"kubernetes.io/projected/72c43d73-596a-48d7-aa34-60c720b92fb6-kube-api-access-khdc5\") pod \"migrator-66df5d496b-vrlcq\" (UID: \"72c43d73-596a-48d7-aa34-60c720b92fb6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq"
Oct 30 20:37:07.894797 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.894772    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdc5\" (UniqueName: \"kubernetes.io/projected/72c43d73-596a-48d7-aa34-60c720b92fb6-kube-api-access-khdc5\") pod \"migrator-66df5d496b-vrlcq\" (UID: \"72c43d73-596a-48d7-aa34-60c720b92fb6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq"
Oct 30 20:37:07.997415 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:07.997391    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf"
Oct 30 20:37:08.011847 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.011660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq"
Oct 30 20:37:08.311627 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.311373    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf"]
Oct 30 20:37:08.334384 ip-10-0-128-161 kubenswrapper[2576]: W1030 20:37:08.334355    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc5b9cd_7d9f_41e9_aa64_aafdc750f62a.slice/crio-1f0e7657fb4e466651c4da1b3cede4610cc2c7ab47b0c4fc3915253fb14f340d WatchSource:0}: Error finding container 1f0e7657fb4e466651c4da1b3cede4610cc2c7ab47b0c4fc3915253fb14f340d: Status 404 returned error can't find the container with id 1f0e7657fb4e466651c4da1b3cede4610cc2c7ab47b0c4fc3915253fb14f340d
Oct 30 20:37:08.339022 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.338993    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq"]
Oct 30 20:37:08.389265 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.387474    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kz7s8"]
Oct 30 20:37:08.413696 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.413613    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:08.428540 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.428508    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7k6rz-jhhmx"]
Oct 30 20:37:08.428675 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.428633    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.441848 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.441811    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:37:08.445758 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.444347    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kz7s8"]
Oct 30 20:37:08.445758 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.444477    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.445758 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.444478    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7k6rz-jhhmx"]
Oct 30 20:37:08.466503 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.466434    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtwx7\""
Oct 30 20:37:08.466503 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.466478    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:37:08.490115 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.490088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-service-ca-bundle\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.490253 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.490139    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-crio-socket\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.490253 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.490164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-data-volume\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.490253 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.490185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.490253 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.490212    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-trusted-ca-bundle\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.490519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.490260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2edfa6-676f-47ab-af10-37be9080c821-archives-path\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.490519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.490283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.490519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.490329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcxr\" (UniqueName: \"kubernetes.io/projected/2d2edfa6-676f-47ab-af10-37be9080c821-kube-api-access-xqcxr\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.490519 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.490372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrlf\" (UniqueName: \"kubernetes.io/projected/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-api-access-hqrlf\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.517827 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.517799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:08.579842 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.579722    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rbf5z" event={"ID":"b3e2fc05-6adb-4c94-a124-aca4483a4e13","Type":"ContainerStarted","Data":"5920e6d7de20c10a6942708c81db189a50991832bcff606858220d8dcd1b7e33"}
Oct 30 20:37:08.583936 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.583875    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9v7g" event={"ID":"93f061fe-5c27-419b-8677-9734d5dce379","Type":"ContainerStarted","Data":"0c6d9a595513ea5aa4eef56e39686ae746114379aaa348b80da1cd83bcdb510f"}
Oct 30 20:37:08.586103 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.586083    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/1.log"
Oct 30 20:37:08.586562 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.586543    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/0.log"
Oct 30 20:37:08.586689 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.586642    2576 generic.go:356] "Generic (PLEG): container finished" podID="ebc0553b-387f-438a-ab73-a053b37abcdd" containerID="f354ab5f6dc4cbe1be1a150d8a98d993f7b1d3c8cf4857d96e6ad15eecfa725f" exitCode=255
Oct 30 20:37:08.587194 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.587151    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" event={"ID":"ebc0553b-387f-438a-ab73-a053b37abcdd","Type":"ContainerDied","Data":"f354ab5f6dc4cbe1be1a150d8a98d993f7b1d3c8cf4857d96e6ad15eecfa725f"}
Oct 30 20:37:08.587302 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.587275    2576 scope.go:122] "RemoveContainer" containerID="0f4f50129487ec968f83493c52eee6745af4d4b6cb7c6bbcdc2adbd910c8c9fc"
Oct 30 20:37:08.587613 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.587590    2576 scope.go:122] "RemoveContainer" containerID="f354ab5f6dc4cbe1be1a150d8a98d993f7b1d3c8cf4857d96e6ad15eecfa725f"
Oct 30 20:37:08.587843 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:08.587821    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8nltv_openshift-console-operator(ebc0553b-387f-438a-ab73-a053b37abcdd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" podUID="ebc0553b-387f-438a-ab73-a053b37abcdd"
Oct 30 20:37:08.590765 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.590747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrlf\" (UniqueName: \"kubernetes.io/projected/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-api-access-hqrlf\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.590880 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.590839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-service-ca-bundle\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.590945 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.590889    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-crio-socket\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.590945 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.590913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-data-volume\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.591046 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.590977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.591046 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.591037    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-trusted-ca-bundle\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.591152 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.591094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2edfa6-676f-47ab-af10-37be9080c821-archives-path\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.591152 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.591117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.591258 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.591210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcxr\" (UniqueName: \"kubernetes.io/projected/2d2edfa6-676f-47ab-af10-37be9080c821-kube-api-access-xqcxr\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.591373 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.591355    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-data-volume\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.591452 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.591380    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-crio-socket\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.592054 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:08.591505    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:37:08.592054 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:08.591560    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:09.091541327 +0000 UTC m=+37.227064466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : configmap "kube-rbac-proxy" not found
Oct 30 20:37:08.592054 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:08.591629    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:08.592054 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:08.591665    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:09.091653201 +0000 UTC m=+37.227176345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:08.592054 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.591980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-service-ca-bundle\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.592356 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.592248    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2" event={"ID":"a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf","Type":"ContainerStarted","Data":"05bee4abc8d334abc1e6c2486179a212b36a9cd8d4f2e53dc7b22481011d587e"}
Oct 30 20:37:08.592504 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.592478    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-trusted-ca-bundle\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.592712 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.592688    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2edfa6-676f-47ab-af10-37be9080c821-archives-path\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.594605 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.594583    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd" event={"ID":"b4376a74-205b-45a1-b005-c59fae4f4201","Type":"ContainerStarted","Data":"191465458b3c2d97e34b6a8014d8204c0eda55bbdb2f5c4475a210695df9a72a"}
Oct 30 20:37:08.597886 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.597842    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq" event={"ID":"72c43d73-596a-48d7-aa34-60c720b92fb6","Type":"ContainerStarted","Data":"8e8ae9789e9c92ac14c322e2538f04efc9662b1e1e7fb068b9994c3b513cec0e"}
Oct 30 20:37:08.601011 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.600929    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf" event={"ID":"2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a","Type":"ContainerStarted","Data":"1f0e7657fb4e466651c4da1b3cede4610cc2c7ab47b0c4fc3915253fb14f340d"}
Oct 30 20:37:08.603891 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.603157    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-rbf5z" podStartSLOduration=343.33270892 podStartE2EDuration="5m48.603142566s" totalImagesPullingTime="5.270433646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:08.602310516 +0000 UTC m=+36.737833670" watchObservedRunningTime="2026-10-30 20:37:08.603142566 +0000 UTC m=+36.738665722"
Oct 30 20:37:08.609693 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.609664    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcxr\" (UniqueName: \"kubernetes.io/projected/2d2edfa6-676f-47ab-af10-37be9080c821-kube-api-access-xqcxr\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.612200 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.612162    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrlf\" (UniqueName: \"kubernetes.io/projected/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-api-access-hqrlf\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:08.614315 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.614193    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:08.642888 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.642825    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd" podStartSLOduration=343.32999993 podStartE2EDuration="5m48.642815745s" totalImagesPullingTime="5.312815815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:08.64201451 +0000 UTC m=+36.777537666" watchObservedRunningTime="2026-10-30 20:37:08.642815745 +0000 UTC m=+36.778338899"
Oct 30 20:37:08.659002 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.658948    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2" podStartSLOduration=343.172629329 podStartE2EDuration="5m48.658928612s" totalImagesPullingTime="5.486299283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:08.657485405 +0000 UTC m=+36.793008561" watchObservedRunningTime="2026-10-30 20:37:08.658928612 +0000 UTC m=+36.794451769"
Oct 30 20:37:08.704785 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.704671    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:08.775162 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.775126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:08.838749 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.838664    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x9v7g" podStartSLOduration=2.629284406 podStartE2EDuration="7.838644402s" totalImagesPullingTime="5.209359996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:08.675090883 +0000 UTC m=+36.810614039" watchObservedRunningTime="2026-10-30 20:37:08.838644402 +0000 UTC m=+36.974167559"
Oct 30 20:37:08.839285 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.839264    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zqt6s"]
Oct 30 20:37:08.856803 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.855688    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:08.861188 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.859103    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zqt6s"]
Oct 30 20:37:08.862465 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.861581    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 20:37:08.862465 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.862201    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 20:37:08.862465 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.862270    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 20:37:08.862465 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.862347    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 20:37:08.862465 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.862427    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-q7pnc\""
Oct 30 20:37:08.894167 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.893809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-key\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:08.894167 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.893877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-cabundle\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:08.894167 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.893905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jfd\" (UniqueName: \"kubernetes.io/projected/1d7877a9-9cc5-4655-9b03-24fd447878ac-kube-api-access-k2jfd\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:08.995724 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.995684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-key\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:08.995951 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.995736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-cabundle\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:08.995951 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.995760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jfd\" (UniqueName: \"kubernetes.io/projected/1d7877a9-9cc5-4655-9b03-24fd447878ac-kube-api-access-k2jfd\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:08.997354 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.997296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-cabundle\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:08.999703 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:08.999677    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-key\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:09.006172 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.006138    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jfd\" (UniqueName: \"kubernetes.io/projected/1d7877a9-9cc5-4655-9b03-24fd447878ac-kube-api-access-k2jfd\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:09.097684 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.096802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:09.097684 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.096913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:09.097684 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.097128    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:37:09.097684 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.097187    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:10.097170915 +0000 UTC m=+38.232694052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : configmap "kube-rbac-proxy" not found
Oct 30 20:37:09.097684 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.097572    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:09.097684 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.097615    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:10.097599797 +0000 UTC m=+38.233122945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:09.102378 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.102353    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7k6rz-jhhmx"]
Oct 30 20:37:09.193907 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.193709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:09.229158 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.229128    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:09.517879 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.517834    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zqt6s"]
Oct 30 20:37:09.607294 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.607215    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s" event={"ID":"1d7877a9-9cc5-4655-9b03-24fd447878ac","Type":"ContainerStarted","Data":"89894ec91743e630b725b87d833a6bf3dc8968b83f1f1bfd6b6825f786df8de9"}
Oct 30 20:37:09.608926 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.608872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:09.609038 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.608931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:09.609038 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.608958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:09.609038 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.609009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:09.609193 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609033    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:09.609193 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609051    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-948b95b59-tdzdl: secret "image-registry-tls" not found
Oct 30 20:37:09.609193 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.609100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:09.609193 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609132    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:37:09.609193 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.609140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:09.609193 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.609171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:09.609193 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609188    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls podName:47697b9d-ed21-4e3b-93b0-320138d4037d nodeName:}" failed. No retries permitted until 2026-10-30 20:37:17.609168497 +0000 UTC m=+45.744691635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cd75b" (UID: "47697b9d-ed21-4e3b-93b0-320138d4037d") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.609236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609260    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609298    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls podName:f7aac07e-1e6f-4f62-897c-0061aa870749 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:17.60928527 +0000 UTC m=+45.744808427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-crfsg" (UID: "f7aac07e-1e6f-4f62-897c-0061aa870749") : secret "samples-operator-tls" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609314    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls podName:4dd510fb-f40e-401c-845f-23b9d750dc68 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:17.609305171 +0000 UTC m=+45.744828311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls") pod "image-registry-948b95b59-tdzdl" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68") : secret "image-registry-tls" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609365    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609365    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:17.609353231 +0000 UTC m=+45.744876368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : configmap references non-existent config key: service-ca.crt
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609030    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609402    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls podName:6140cde0-b52b-48e1-a1d6-bcac35086922 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:17.609391542 +0000 UTC m=+45.744914680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls") pod "dns-default-nfvzn" (UID: "6140cde0-b52b-48e1-a1d6-bcac35086922") : secret "dns-default-metrics-tls" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609422    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609431    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:17.609417462 +0000 UTC m=+45.744940629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : secret "router-metrics-certs-default" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609434    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfdcc7f98-zlptz: secret "image-registry-tls" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609455    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609486    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls podName:204ba748-a514-4772-9fba-dfafd3eddcf7 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:17.609472112 +0000 UTC m=+45.744995247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls") pod "image-registry-5bfdcc7f98-zlptz" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7") : secret "image-registry-tls" not found
Oct 30 20:37:09.609524 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.609503    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert podName:734323fa-6946-426b-9e06-396cb4e7b48f nodeName:}" failed. No retries permitted until 2026-10-30 20:37:17.609494502 +0000 UTC m=+45.745017637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wt7nr" (UID: "734323fa-6946-426b-9e06-396cb4e7b48f") : secret "networking-console-plugin-cert" not found
Oct 30 20:37:09.613778 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.613736    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/1.log"
Oct 30 20:37:09.614224 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.614201    2576 scope.go:122] "RemoveContainer" containerID="f354ab5f6dc4cbe1be1a150d8a98d993f7b1d3c8cf4857d96e6ad15eecfa725f"
Oct 30 20:37:09.614441 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:09.614416    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8nltv_openshift-console-operator(ebc0553b-387f-438a-ab73-a053b37abcdd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" podUID="ebc0553b-387f-438a-ab73-a053b37abcdd"
Oct 30 20:37:09.619020 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.618992    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7k6rz-jhhmx" event={"ID":"2d2edfa6-676f-47ab-af10-37be9080c821","Type":"ContainerStarted","Data":"1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef"}
Oct 30 20:37:09.619143 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.619059    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7k6rz-jhhmx" event={"ID":"2d2edfa6-676f-47ab-af10-37be9080c821","Type":"ContainerStarted","Data":"5b7e77e725cde65cecf7918764087c2c53584f845186c677033925e14d6db3b7"}
Oct 30 20:37:09.818627 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:09.818359    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.115518 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.115443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:10.115647 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.115528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:10.115647 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:10.115606    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:10.115758 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:10.115681    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:12.115655562 +0000 UTC m=+40.251178699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:10.115758 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:10.115702    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:37:10.115758 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:10.115742    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:12.115729501 +0000 UTC m=+40.251252653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : configmap "kube-rbac-proxy" not found
Oct 30 20:37:10.329875 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.323297    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.423212 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.423162    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.522906 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.522874    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.624468 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.624435    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq" event={"ID":"72c43d73-596a-48d7-aa34-60c720b92fb6","Type":"ContainerStarted","Data":"773ffd41b47a22d5fd9ca335d88f9a7b82057c5d4475a9ce78ce7499a2084bcc"}
Oct 30 20:37:10.626606 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.626584    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf" event={"ID":"2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a","Type":"ContainerStarted","Data":"03fc93ad4988d52d2da7eb0f20210ce085e6da3ae7f2684fbd09c7e1dd1d4e82"}
Oct 30 20:37:10.629495 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.629429    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s" event={"ID":"1d7877a9-9cc5-4655-9b03-24fd447878ac","Type":"ContainerStarted","Data":"ad09c2df0017473eac006ae6556a4ef63379e3e5215a4d3bfa9c1d26cc58aecf"}
Oct 30 20:37:10.634957 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.633587    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.646623 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.646571    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7k6rz-jhhmx" podStartSLOduration=2.646552909 podStartE2EDuration="2.646552909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:09.650708869 +0000 UTC m=+37.786232023" watchObservedRunningTime="2026-10-30 20:37:10.646552909 +0000 UTC m=+38.782076067"
Oct 30 20:37:10.648081 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.647269    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s" podStartSLOduration=2.647257131 podStartE2EDuration="2.647257131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:10.646026682 +0000 UTC m=+38.781549840" watchObservedRunningTime="2026-10-30 20:37:10.647257131 +0000 UTC m=+38.782780289"
Oct 30 20:37:10.723649 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.723568    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.831425 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:10.831391    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:11.232557 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:11.232530    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4mv46_7389168a-7f19-48df-98f8-6d6d20892416/dns-node-resolver/0.log"
Oct 30 20:37:11.634315 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:11.634282    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lfl79" event={"ID":"2fc2fd6c-3d77-4446-9402-cf73cdd3497a","Type":"ContainerStarted","Data":"5bc9ff6e16c4ada4efe41dc8d79af05998715e344027c68069cde949ad200528"}
Oct 30 20:37:11.634481 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:11.634463    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:37:11.650462 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:11.650418    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lfl79" podStartSLOduration=35.713827131 podStartE2EDuration="39.650406201s" totalImagesPullingTime="3.93657907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:11.649279928 +0000 UTC m=+39.784803085" watchObservedRunningTime="2026-10-30 20:37:11.650406201 +0000 UTC m=+39.785929356"
Oct 30 20:37:11.666832 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:11.666786    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq" podStartSLOduration=2.48669953 podStartE2EDuration="4.666770531s" totalImagesPullingTime="2.180071001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:11.665189927 +0000 UTC m=+39.800713082" watchObservedRunningTime="2026-10-30 20:37:11.666770531 +0000 UTC m=+39.802293692"
Oct 30 20:37:11.681197 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:11.681158    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf" podStartSLOduration=2.4935147730000002 podStartE2EDuration="4.681145657s" totalImagesPullingTime="2.187630884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:11.679630563 +0000 UTC m=+39.815153718" watchObservedRunningTime="2026-10-30 20:37:11.681145657 +0000 UTC m=+39.816668812"
Oct 30 20:37:11.831726 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:11.831697    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sq5lx_58b187ea-dead-4a37-8a52-5e447783d815/node-ca/0.log"
Oct 30 20:37:11.950721 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:11.950627    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:12.123131 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.123091    2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:12.123273 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.123159    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:12.123576 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.123563    2576 scope.go:122] "RemoveContainer" containerID="f354ab5f6dc4cbe1be1a150d8a98d993f7b1d3c8cf4857d96e6ad15eecfa725f"
Oct 30 20:37:12.123805 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:12.123783    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8nltv_openshift-console-operator(ebc0553b-387f-438a-ab73-a053b37abcdd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" podUID="ebc0553b-387f-438a-ab73-a053b37abcdd"
Oct 30 20:37:12.139226 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.139196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:12.139341 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.139285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:12.139394 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:12.139355    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:12.139447 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:12.139417    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:16.139401427 +0000 UTC m=+44.274924567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:12.139447 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:12.139423    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:37:12.139555 ip-10-0-128-161 kubenswrapper[2576]: E1030 20:37:12.139482    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:16.139465301 +0000 UTC m=+44.274988449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : configmap "kube-rbac-proxy" not found
Oct 30 20:37:12.145803 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.145774    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:12.232108 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.231913    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9v7g_93f061fe-5c27-419b-8677-9734d5dce379/serve-healthcheck-canary/0.log"
Oct 30 20:37:12.352242 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.352211    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:12.441186 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.440975    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vrlcq_72c43d73-596a-48d7-aa34-60c720b92fb6/migrator/0.log"
Oct 30 20:37:12.545471 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.545381    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:12.634054 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.634004    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-xpdvf_2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a/migrator/0.log"
Oct 30 20:37:12.748457 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.748422    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:12.833932 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.833824    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n_395a48bf-cfce-443e-be13-b2939a9b9b8a/kube-storage-version-migrator-operator/0.log"
Oct 30 20:37:12.848041 ip-10-0-128-161 kubenswrapper[2576]: I1030 20:37:12.848004    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:13.426652 ip-10-0-128-161 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 20:37:13.444484 ip-10-0-128-161 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 20:37:13.444650 ip-10-0-128-161 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 20:37:13.444804 ip-10-0-128-161 systemd[1]: kubelet.service: Consumed 3.204s CPU time.
Oct 30 20:37:13.454434 ip-10-0-128-161 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:37:13.558047 ip-10-0-128-161 kubenswrapper[4810]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:37:13.558047 ip-10-0-128-161 kubenswrapper[4810]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:37:13.558047 ip-10-0-128-161 kubenswrapper[4810]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:37:13.558047 ip-10-0-128-161 kubenswrapper[4810]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:37:13.560277 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560201    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:13.560277 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560273    4810 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:13.560277 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560278    4810 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:13.560277 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560281    4810 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560304    4810 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560309    4810 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560312    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560316    4810 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560318    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560321    4810 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560324    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560327    4810 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560329    4810 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560332    4810 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560335    4810 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560337    4810 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560340    4810 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560343    4810 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560353    4810 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560357    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560360    4810 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560363    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560366    4810 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:13.560443 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560369    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560389    4810 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560395    4810 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560398    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560402    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560410    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560413    4810 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560416    4810 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560418    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560421    4810 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560424    4810 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560427    4810 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560429    4810 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560432    4810 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560434    4810 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560437    4810 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560439    4810 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560442    4810 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560445    4810 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:13.560941 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560448    4810 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560451    4810 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560454    4810 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560456    4810 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560459    4810 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560461    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560464    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560466    4810 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560469    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560471    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560474    4810 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560477    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560479    4810 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560482    4810 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560484    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560487    4810 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560490    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560492    4810 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560496    4810 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560499    4810 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:13.561413 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560502    4810 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560505    4810 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560507    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560510    4810 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560512    4810 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560515    4810 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560517    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560520    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560522    4810 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560525    4810 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560528    4810 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560530    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560533    4810 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560535    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560538    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560541    4810 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560543    4810 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560546    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560548    4810 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:13.561915 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560551    4810 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560554    4810 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560557    4810 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560561    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560564    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560567    4810 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560569    4810 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560572    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560575    4810 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560578    4810 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560581    4810 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560583    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560586    4810 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560595    4810 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560598    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560601    4810 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560603    4810 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560606    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560609    4810 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:13.562415 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560611    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560614    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560618    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560620    4810 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560623    4810 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.560625    4810 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561403    4810 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561410    4810 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561413    4810 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561416    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561419    4810 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561421    4810 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561424    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561428    4810 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561431    4810 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561434    4810 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561436    4810 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561439    4810 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561441    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561445    4810 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:13.562939 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561447    4810 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561450    4810 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561453    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561455    4810 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561460    4810 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561464    4810 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561468    4810 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561479    4810 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561483    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561485    4810 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561488    4810 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561491    4810 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561494    4810 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561497    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561500    4810 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561503    4810 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561506    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561508    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561513    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:13.563548 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561515    4810 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561518    4810 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561520    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561523    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561526    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561543    4810 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561548    4810 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561551    4810 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561553    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561556    4810 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561559    4810 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561561    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561564    4810 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561567    4810 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561571    4810 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561574    4810 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561578    4810 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561581    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561584    4810 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:13.564149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561587    4810 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561589    4810 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561597    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561600    4810 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561603    4810 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561605    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561608    4810 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561610    4810 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561613    4810 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561615    4810 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561618    4810 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561620    4810 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561623    4810 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561631    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561634    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561636    4810 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561639    4810 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561642    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561645    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:13.564673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561647    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561650    4810 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561653    4810 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561655    4810 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561658    4810 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561661    4810 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561663    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561666    4810 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561669    4810 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561671    4810 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561674    4810 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561676    4810 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561678    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561681    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561683    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561686    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561693    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561696    4810 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561699    4810 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:13.565155 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561702    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561705    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561707    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561709    4810 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561712    4810 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561714    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561717    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561720    4810 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561722    4810 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561725    4810 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561728    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561731    4810 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561734    4810 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561737    4810 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561739    4810 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.561742    4810 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564085    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564098    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564102    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564105    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:13.565645 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564108    4810 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564110    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564113    4810 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564118    4810 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564121    4810 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564124    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564127    4810 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564130    4810 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564133    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564137    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564141    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564146    4810 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564151    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564154    4810 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564157    4810 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564160    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564163    4810 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564165    4810 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564168    4810 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564171    4810 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:13.566144 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564174    4810 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564177    4810 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564182    4810 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564186    4810 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564190    4810 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564196    4810 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564199    4810 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564204    4810 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564212    4810 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564217    4810 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564221    4810 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564226    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564230    4810 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564233    4810 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564236    4810 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564238    4810 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564241    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564244    4810 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564246    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:13.566759 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564249    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564252    4810 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564254    4810 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564257    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564259    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564262    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564265    4810 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564274    4810 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564281    4810 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564285    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564288    4810 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564291    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564294    4810 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564296    4810 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564299    4810 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564301    4810 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564304    4810 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564308    4810 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564312    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:13.567268 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564315    4810 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564327    4810 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564331    4810 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564334    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564337    4810 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564340    4810 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564343    4810 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564347    4810 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564349    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564351    4810 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564354    4810 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564357    4810 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564360    4810 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564362    4810 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564365    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564367    4810 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564370    4810 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564372    4810 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564375    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564378    4810 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:13.567734 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564380    4810 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564384    4810 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564387    4810 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564389    4810 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564392    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564395    4810 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564397    4810 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564400    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564402    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564405    4810 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564408    4810 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564410    4810 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564412    4810 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564415    4810 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564418    4810 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564420    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564423    4810 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564427    4810 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564429    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564432    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:13.568411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564435    4810 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564437    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564440    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.564442    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564536    4810 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564543    4810 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564550    4810 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564555    4810 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564559    4810 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564563    4810 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564567    4810 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564571    4810 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564574    4810 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564577    4810 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564581    4810 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564584    4810 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564587    4810 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564590    4810 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564593    4810 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564596    4810 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564599    4810 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564602    4810 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564607    4810 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:37:13.569015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564609    4810 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564613    4810 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564618    4810 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564622    4810 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564626    4810 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564629    4810 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564632    4810 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564635    4810 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564638    4810 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564642    4810 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564645    4810 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564648    4810 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564651    4810 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564655    4810 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564658    4810 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564661    4810 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564664    4810 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564668    4810 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564671    4810 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564676    4810 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564679    4810 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564682    4810 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564686    4810 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564689    4810 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:37:13.569589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564694    4810 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564697    4810 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564701    4810 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564704    4810 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564707    4810 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564710    4810 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564712    4810 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564716    4810 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564719    4810 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564722    4810 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564728    4810 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564732    4810 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564736    4810 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564739    4810 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564742    4810 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564745    4810 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564748    4810 flags.go:64] FLAG: --help="false"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564754    4810 flags.go:64] FLAG: --hostname-override="ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564758    4810 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564761    4810 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564764    4810 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564767    4810 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564771    4810 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:37:13.570521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564774    4810 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564777    4810 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564780    4810 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564783    4810 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564786    4810 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564789    4810 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564792    4810 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564795    4810 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564798    4810 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564801    4810 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564804    4810 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564807    4810 flags.go:64] FLAG: --lock-file=""
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564810    4810 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564813    4810 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564816    4810 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564826    4810 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564829    4810 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564833    4810 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564836    4810 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564840    4810 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564844    4810 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564847    4810 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564850    4810 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564868    4810 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564872    4810 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:37:13.571502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564876    4810 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564879    4810 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564882    4810 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564885    4810 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564888    4810 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564891    4810 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564894    4810 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564897    4810 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564906    4810 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564909    4810 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564912    4810 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564915    4810 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564918    4810 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564921    4810 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564925    4810 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564928    4810 flags.go:64] FLAG: --port="10250"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564931    4810 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564933    4810 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0056c41d4d4faca06"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564938    4810 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564941    4810 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564944    4810 flags.go:64] FLAG: --register-node="true"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564950    4810 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564958    4810 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564961    4810 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564965    4810 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564968    4810 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:37:13.572616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564973    4810 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564977    4810 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564980    4810 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564983    4810 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564986    4810 flags.go:64] FLAG: --runonce="false"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564989    4810 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564992    4810 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564995    4810 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.564998    4810 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565001    4810 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565004    4810 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565008    4810 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565011    4810 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565014    4810 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565016    4810 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565020    4810 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565023    4810 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565026    4810 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565029    4810 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565032    4810 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565039    4810 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565041    4810 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565044    4810 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565049    4810 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565051    4810 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:37:13.573699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565055    4810 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565058    4810 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565062    4810 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565066    4810 flags.go:64] FLAG: --v="2"
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565070    4810 flags.go:64] FLAG: --version="false"
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565074    4810 flags.go:64] FLAG: --vmodule=""
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565078    4810 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565082    4810 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565212    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565216    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565219    4810 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565222    4810 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565225    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565228    4810 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565230    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565233    4810 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565235    4810 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565238    4810 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565241    4810 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565243    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565246    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:13.574639 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565248    4810 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565251    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565253    4810 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565256    4810 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565258    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565262    4810 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565264    4810 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565267    4810 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565269    4810 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565272    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565275    4810 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565277    4810 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565280    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565284    4810 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565288    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565290    4810 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565293    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565295    4810 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565299    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565303    4810 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:13.575177 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565306    4810 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565309    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565312    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565315    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565318    4810 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565321    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565323    4810 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565326    4810 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565328    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565331    4810 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565334    4810 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565336    4810 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565339    4810 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565342    4810 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565344    4810 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565347    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565349    4810 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565352    4810 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565354    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565357    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:13.575657 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565360    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565362    4810 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565365    4810 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565367    4810 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565370    4810 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565374    4810 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565377    4810 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565380    4810 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565382    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565385    4810 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565389    4810 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565392    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565394    4810 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565397    4810 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565400    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565402    4810 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565405    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565407    4810 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565410    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:13.576168 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565412    4810 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565415    4810 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565418    4810 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565420    4810 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565423    4810 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565425    4810 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565428    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565431    4810 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565433    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565436    4810 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565438    4810 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565441    4810 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565444    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565446    4810 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565449    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565451    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565454    4810 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565457    4810 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565460    4810 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:13.576624 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565468    4810 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565473    4810 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565477    4810 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565480    4810 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565484    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565486    4810 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565489    4810 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565492    4810 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565495    4810 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565497    4810 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565500    4810 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565502    4810 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565505    4810 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565507    4810 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565510    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:13.577166 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.565536    4810 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565581    4810 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565586    4810 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565589    4810 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565592    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565594    4810 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565597    4810 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565599    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565602    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565604    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565607    4810 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565611    4810 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565616    4810 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565619    4810 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565623    4810 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565627    4810 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565633    4810 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565637    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565641    4810 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565644    4810 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:13.577641 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565647    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565650    4810 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565654    4810 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565657    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565660    4810 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565663    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565665    4810 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565668    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565670    4810 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565673    4810 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565675    4810 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565678    4810 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565680    4810 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565683    4810 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565685    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565688    4810 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565691    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565694    4810 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565696    4810 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565699    4810 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:13.578196 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565701    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565704    4810 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565706    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565709    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565711    4810 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565714    4810 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565717    4810 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565719    4810 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565725    4810 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565729    4810 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565732    4810 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565736    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565739    4810 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565742    4810 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565746    4810 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565748    4810 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565751    4810 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565753    4810 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565756    4810 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:13.578787 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565759    4810 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565761    4810 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565764    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565766    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565769    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565771    4810 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565774    4810 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565776    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565779    4810 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565781    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565784    4810 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565786    4810 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565789    4810 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565791    4810 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565794    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565796    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565798    4810 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565801    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565803    4810 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:13.579290 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565806    4810 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565808    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565813    4810 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565816    4810 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565818    4810 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565820    4810 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565823    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565826    4810 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565829    4810 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565832    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565835    4810 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565837    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565840    4810 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565843    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565845    4810 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565848    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565852    4810 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565880    4810 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565884    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565887    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:13.579769 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565890    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565893    4810 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565895    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565898    4810 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565900    4810 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565903    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565905    4810 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565908    4810 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.565910    4810 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.571506    4810 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.571522    4810 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.571541    4810 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.571546    4810 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571609    4810 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571614    4810 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571618    4810 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571621    4810 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571624    4810 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571627    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571629    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:13.580260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571633    4810 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571635    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571638    4810 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571641    4810 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571647    4810 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571651    4810 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571657    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571661    4810 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571665    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571668    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571671    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571683    4810 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571686    4810 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571688    4810 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571691    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571694    4810 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571696    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571699    4810 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571702    4810 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:13.580738 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571704    4810 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571707    4810 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571709    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571713    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571716    4810 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571719    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571721    4810 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571726    4810 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571730    4810 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571734    4810 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571738    4810 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571743    4810 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571747    4810 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571751    4810 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571755    4810 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571759    4810 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571761    4810 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571764    4810 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571767    4810 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571770    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:13.581263 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571773    4810 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571776    4810 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571778    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571781    4810 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571783    4810 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571786    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571788    4810 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571791    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571794    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571797    4810 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571800    4810 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571802    4810 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571805    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571808    4810 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571810    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571813    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571817    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571822    4810 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571826    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571831    4810 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:13.581744 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571835    4810 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571838    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571841    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571843    4810 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571846    4810 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571848    4810 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571851    4810 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571870    4810 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571873    4810 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571876    4810 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571879    4810 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571882    4810 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571884    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571887    4810 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571890    4810 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571893    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571896    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571900    4810 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571904    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571909    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:13.582242 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571913    4810 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571918    4810 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571921    4810 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571923    4810 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571927    4810 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571932    4810 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571936    4810 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571940    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571944    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571948    4810 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571952    4810 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571955    4810 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571958    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571961    4810 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571963    4810 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571967    4810 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571969    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571973    4810 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571976    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:13.582711 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.571980    4810 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.572013    4810 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572161    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572167    4810 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572170    4810 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572173    4810 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572175    4810 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572178    4810 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572181    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572184    4810 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572186    4810 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572189    4810 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572192    4810 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572194    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572197    4810 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572200    4810 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:13.583264 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572202    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572205    4810 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572208    4810 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572210    4810 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572213    4810 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572215    4810 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572218    4810 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572222    4810 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572226    4810 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572230    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572235    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572239    4810 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572242    4810 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572245    4810 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572247    4810 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572251    4810 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572253    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572257    4810 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572259    4810 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572262    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:13.583658 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572264    4810 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572266    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572269    4810 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572272    4810 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572274    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572276    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572279    4810 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572281    4810 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572284    4810 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572287    4810 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572289    4810 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572291    4810 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572294    4810 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572297    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572299    4810 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572302    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572306    4810 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572310    4810 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572314    4810 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572319    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:13.584220 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572324    4810 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572327    4810 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572330    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572332    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572335    4810 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572337    4810 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572340    4810 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572342    4810 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572345    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572348    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572350    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572353    4810 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572356    4810 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572358    4810 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572361    4810 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572363    4810 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572366    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572368    4810 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572371    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572374    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:13.584739 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572376    4810 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572379    4810 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572381    4810 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572384    4810 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572389    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572393    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572397    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572401    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572405    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572408    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572410    4810 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572413    4810 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572415    4810 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572418    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572421    4810 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572425    4810 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572429    4810 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572433    4810 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572436    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:13.585229 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572440    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572443    4810 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572446    4810 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572449    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572451    4810 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572454    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572457    4810 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572460    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572462    4810 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572465    4810 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572468    4810 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572473    4810 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572477    4810 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.572503    4810 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572548    4810 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572554    4810 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:13.585692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572559    4810 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572564    4810 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572568    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572571    4810 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572574    4810 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572577    4810 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572580    4810 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572583    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572585    4810 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572588    4810 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572591    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572594    4810 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572597    4810 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572599    4810 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572602    4810 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572604    4810 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572607    4810 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572610    4810 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572613    4810 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:13.586127 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572616    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572618    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572621    4810 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572624    4810 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572626    4810 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572630    4810 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572635    4810 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572639    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572644    4810 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572648    4810 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572650    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572653    4810 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572656    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572658    4810 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572661    4810 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572663    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572665    4810 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572668    4810 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572671    4810 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572673    4810 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:13.586568 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572676    4810 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572678    4810 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572681    4810 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572683    4810 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572686    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572689    4810 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572692    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572694    4810 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572697    4810 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572700    4810 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572706    4810 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572709    4810 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572713    4810 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572718    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572722    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572726    4810 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572730    4810 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572733    4810 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572736    4810 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572739    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:13.587058 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572741    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572744    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572747    4810 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572750    4810 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572752    4810 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572755    4810 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572758    4810 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572761    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572763    4810 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572766    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572769    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572771    4810 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572774    4810 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572777    4810 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572779    4810 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572782    4810 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572784    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572787    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572790    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572795    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:13.587543 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572799    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572803    4810 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572809    4810 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572812    4810 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572815    4810 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572817    4810 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572821    4810 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572826    4810 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572831    4810 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572833    4810 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572836    4810 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572839    4810 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572841    4810 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572844    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572847    4810 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572849    4810 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572852    4810 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572870    4810 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:13.588085 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572874    4810 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572878    4810 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572885    4810 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572891    4810 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572895    4810 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572898    4810 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:13.572901    4810 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.572956    4810 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.573986    4810 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.574055    4810 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.574238    4810 server.go:1014] "Starting client certificate rotation"
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.574375    4810 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.574417    4810 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:31:32 +0000 UTC" deadline="2028-07-20 15:49:45.337509466 +0000 UTC"
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.574444    4810 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15091h12m31.763066724s"
Oct 30 20:37:13.588517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.574726    4810 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:37:13.594216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.594200    4810 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:37:13.595716 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.595702    4810 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:37:13.597395 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.597381    4810 log.go:25] "Validated CRI v1 image API"
Oct 30 20:37:13.598463 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.598449    4810 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:37:13.601282 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.601260    4810 fs.go:135] Filesystem UUIDs: map[02c9c38f-07fd-4d4b-8c39-5600dc070a2f:/dev/nvme0n1p4 76d67411-3c77-4ee0-8ae4-6cbebfe6a044:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 20:37:13.601535 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.601278    4810 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fc7e09d5a272e8b6c1a912933f9ed70ff39777b9df34e476cda8442364a703b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fc7e09d5a272e8b6c1a912933f9ed70ff39777b9df34e476cda8442364a703b/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d4395f4806e92dfb0cbfee2babccbd31159975836f2db0df3a1548716eb312f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d4395f4806e92dfb0cbfee2babccbd31159975836f2db0df3a1548716eb312f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dad5510556bc61a7a7c6d5b3110720bcfe7b95fc35c5038fa78e3c1d72256a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dad5510556bc61a7a7c6d5b3110720bcfe7b95fc35c5038fa78e3c1d72256a4/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f0e7657fb4e466651c4da1b3cede4610cc2c7ab47b0c4fc3915253fb14f340d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f0e7657fb4e466651c4da1b3cede4610cc2c7ab47b0c4fc3915253fb14f340d/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27c3a01f6e08f36046a0ba85282af82e507a7c028a684a99bdd2e8c672e8a2d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27c3a01f6e08f36046a0ba85282af82e507a7c028a684a99bdd2e8c672e8a2d7/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/450407d6c8501801d763d37fa3ef48d218689f995b03dc5ed8bc4fce95a3e554/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/450407d6c8501801d763d37fa3ef48d218689f995b03dc5ed8bc4fce95a3e554/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d9649e08d287f9cf3755bdc1e0fd411c4abdfd438f4612f03ff2217c604f5c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d9649e08d287f9cf3755bdc1e0fd411c4abdfd438f4612f03ff2217c604f5c2/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4eccb371c44a504ecceba60e71d99b241c6719656d1e3153e55d3e4c58cdc01f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4eccb371c44a504ecceba60e71d99b241c6719656d1e3153e55d3e4c58cdc01f/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/591c79834fe538885aa02a531b5fa1849744a49dda1c869cca457167cd429429/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/591c79834fe538885aa02a531b5fa1849744a49dda1c869cca457167cd429429/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b7e77e725cde65cecf7918764087c2c53584f845186c677033925e14d6db3b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b7e77e725cde65cecf7918764087c2c53584f845186c677033925e14d6db3b7/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b23ef6f859a17d83bde3e36d35d2ae515a76bfea55f5c2762c57472d70669a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b23ef6f859a17d83bde3e36d35d2ae515a76bfea55f5c2762c57472d70669a2/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b2528cedc5eec4e212563e57e0ff38931265057f3bf59f9d1d47d51a42f2284/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b2528cedc5eec4e212563e57e0ff38931265057f3bf59f9d1d47d51a42f2284/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ed8f5dbe5f679f34e4c7e968461203bd8f366a76aebfff9e050a3c135e76bc1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ed8f5dbe5f679f34e4c7e968461203bd8f366a76aebfff9e050a3c135e76bc1/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8463e4ae87cc0710ca9aad3fcffda20a840c5a06db2a0c113445af48fe880008/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8463e4ae87cc0710ca9aad3fcffda20a840c5a06db2a0c113445af48fe880008/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84c4b87f449036eaa626d4e1149ae5e7c367127b152e827259d9f91129c5a168/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84c4b87f449036eaa626d4e1149ae5e7c367127b152e827259d9f91129c5a168/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89894ec91743e630b725b87d833a6bf3dc8968b83f1f1bfd6b6825f786df8de9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89894ec91743e630b725b87d833a6bf3dc8968b83f1f1bfd6b6825f786df8de9/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e8ae9789e9c92ac14c322e2538f04efc9662b1e1e7fb068b9994c3b513cec0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e8ae9789e9c92ac14c322e2538f04efc9662b1e1e7fb068b9994c3b513cec0e/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3a0c9d437acf8b3dae52da884f0c764c3f8f8b01bf4947dd9a149c635bc1584/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3a0c9d437acf8b3dae52da884f0c764c3f8f8b01bf4947dd9a149c635bc1584/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b53bf2d99bd9bb10291cc1130c05774b10c78c543f5a74db35199f5b662b06d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b53bf2d99bd9bb10291cc1130c05774b10c78c543f5a74db35199f5b662b06d5/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1e310a1152b31ec7ee355d51efa554069bcd8a6fb1c07c58982aa4264b9b55a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1e310a1152b31ec7ee355d51efa554069bcd8a6fb1c07c58982aa4264b9b55a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5f43122589f19a195e82f24618908f0f05878856b1a3e1d4bb2f18609ecfc7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5f43122589f19a195e82f24618908f0f05878856b1a3e1d4bb2f18609ecfc7d/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0c8d484c3330acc4ec03058786ab4f68a8cd5fbde42e9da92f9bf28e19f3ec6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0c8d484c3330acc4ec03058786ab4f68a8cd5fbde42e9da92f9bf28e19f3ec6/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e17ab44add68a3afb1672716b073e1a15d617e833b45b102dde94aa0d60f74e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e17ab44add68a3afb1672716b073e1a15d617e833b45b102dde94aa0d60f74e7/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e454def1ef9c855b78b8c4a7113721c71551983116c56705ddb8cd79a617fd43/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e454def1ef9c855b78b8c4a7113721c71551983116c56705ddb8cd79a617fd43/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f326d3df10a2dc09503dae8b594c653b2397e25df090e25d839dac4162fb7ef9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f326d3df10a2dc09503dae8b594c653b2397e25df090e25d839dac4162fb7ef9/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd6a2570855240aa73ed8dc4ce409b82a7a0eba2fc5762c15c3c61341c0921be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd6a2570855240aa73ed8dc4ce409b82a7a0eba2fc5762c15c3c61341c0921be/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d7898c1-9d6c-4382-baa2-e6baae34941a/volumes/kubernetes.io~projected/kube-api-access-b94nz:{mountpoint:/var/lib/kubelet/pods/0d7898c1-9d6c-4382-baa2-e6baae34941a/volumes/kubernetes.io~projected/kube-api-access-b94nz major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d7877a9-9cc5-4655-9b03-24fd447878ac/volumes/kubernetes.io~projected/kube-api-access-k2jfd:{mountpoint:/var/lib/kubelet/pods/1d7877a9-9cc5-4655-9b03-24fd447878ac/volumes/kubernetes.io~projected/kube-api-access-k2jfd major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d7877a9-9cc5-4655-9b03-24fd447878ac/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/1d7877a9-9cc5-4655-9b03-24fd447878ac/volumes/kubernetes.io~secret/signing-key major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~projected/kube-api-access-v7b7j:{mountpoint:/var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~projected/kube-api-access-v7b7j major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ba0647d-e918-4bc7-843b-205e600536ec/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2ba0647d-e918-4bc7-843b-205e600536ec/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ba0647d-e918-4bc7-843b-205e600536ec/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2ba0647d-e918-4bc7-843b-205e600536ec/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ba0647d-e918-4bc7-843b-205e600536ec/volumes/kubernetes.io~projected/kube-api-access-2qgww:{mountpoint:/var/lib/kubelet/pods/2ba0647d-e918-4bc7-843b-205e600536ec/volumes/kubernetes.io~projected/kube-api-access-2qgww major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a/volumes/kubernetes.io~projected/kube-api-access-wxfsd:{mountpoint:/var/lib/kubelet/pods/2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a/volumes/kubernetes.io~projected/kube-api-access-wxfsd major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d2edfa6-676f-47ab-af10-37be9080c821/volumes/kubernetes.io~projected/kube-api-access-xqcxr:{mountpoint:/var/lib/kubelet/pods/2d2edfa6-676f-47ab-af10-37be9080c821/volumes/kubernetes.io~projected/kube-api-access-xqcxr major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fc2fd6c-3d77-4446-9402-cf73cdd3497a/volumes/kubernetes.io~projected/kube-api-access-m8ksf:{mountpoint:/var/lib/kubelet/pods/2fc2fd6c-3d77-4446-9402-cf73cdd3497a/volumes/kubernetes.io~projected/kube-api-access-m8ksf major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/395a48bf-cfce-443e-be13-b2939a9b9b8a/volumes/kubernetes.io~projected/kube-api-access-khjfm:{mountpoint:/var/lib/kubelet/pods/395a48bf-cfce-443e-be13-b2939a9b9b8a/volumes/kubernetes.io~projected/kube-api-access-khjfm major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/395a48bf-cfce-443e-be13-b2939a9b9b8a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/395a48bf-cfce-443e-be13-b2939a9b9b8a/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3efcea4c-b6b6-47a5-be1d-c61a83bbe907/volumes/kubernetes.io~projected/kube-api-access-qnxvc:{mountpoint:/var/lib/kubelet/pods/3efcea4c-b6b6-47a5-be1d-c61a83bbe907/volumes/kubernetes.io~projected/kube-api-access-qnxvc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47697b9d-ed21-4e3b-93b0-320138d4037d/volumes/kubernetes.io~projected/kube-api-access-dtw88:{mountpoint:/var/lib/kubelet/pods/47697b9d-ed21-4e3b-93b0-320138d4037d/volumes/kubernetes.io~projected/kube-api-access-dtw88 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~projected/kube-api-access-jvtbb:{mountpoint:/var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~projected/kube-api-access-jvtbb major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51d19e04-9e59-4f4f-be5c-9900cca6d655/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/51d19e04-9e59-4f4f-be5c-9900cca6d655/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/585e93ee-0fc2-4461-b5ad-e9e0f50afc06/volumes/kubernetes.io~projected/kube-api-access-hqrlf:{mountpoint:/var/lib/kubelet/pods/585e93ee-0fc2-4461-b5ad-e9e0f50afc06/volumes/kubernetes.io~projected/kube-api-access-hqrlf major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58b187ea-dead-4a37-8a52-5e447783d815/volumes/kubernetes.io~projected/kube-api-access-gpbbp:{mountpoint:/var/lib/kubelet/pods/58b187ea-dead-4a37-8a52-5e447783d815/volumes/kubernetes.io~projected/kube-api-access-gpbbp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6140cde0-b52b-48e1-a1d6-bcac35086922/volumes/kubernetes.io~projected/kube-api-access-wmcl5:{mountpoint:/var/lib/kubelet/pods/6140cde0-b52b-48e1-a1d6-bcac35086922/volumes/kubernetes.io~projected/kube-api-access-wmcl5 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/volumes/kubernetes.io~projected/kube-api-access-plcbb:{mountpoint:/var/lib/kubelet/pods/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/volumes/kubernetes.io~projected/kube-api-access-plcbb major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/volumes/kubernetes.io~secret/default-certificate major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72c43d73-596a-48d7-aa34-60c720b92fb6/volumes/kubernetes.io~projected/kube-api-access-khdc5:{mountpoint:/var/lib/kubelet/pods/72c43d73-596a-48d7-aa34-60c720b92fb6/volumes/kubernetes.io~projected/kube-api-access-khdc5 major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7389168a-7f19-48df-98f8-6d6d20892416/volumes/kubernetes.io~projected/kube-api-access-2bv5x:{mountpoint:/var/lib/kubelet/pods/7389168a-7f19-48df-98f8-6d6d20892416/volumes/kubernetes.io~projected/kube-api-access-2bv5x major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73fa532a-7546-4e60-8f8c-901cebfd7f07/volumes/kubernetes.io~projected/kube-api-access-7zlkc:{mountpoint:/var/lib/kubelet/pods/73fa532a-7546-4e60-8f8c-901cebfd7f07/volumes/kubernetes.io~projected/kube-api-access-7zlkc major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fc47c20-4a62-4167-9334-4d7b1cadb5b5/volumes/kubernetes.io~projected/kube-api-access-dhwsm:{mountpoint:/var/lib/kubelet/pods/7fc47c20-4a62-4167-9334-4d7b1cadb5b5/volumes/kubernetes.io~projected/kube-api-access-dhwsm major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8902c3bd-bf64-48e1-b113-d42ed8456348/volumes/kubernetes.io~projected/kube-api-access-xph2w:{mountpoint:/var/lib/kubelet/pods/8902c3bd-bf64-48e1-b113-d42ed8456348/volumes/kubernetes.io~projected/kube-api-access-xph2w major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a707e7f-24ce-4322-b5ad-ffaaa21dc23c/volumes/kubernetes.io~projected/kube-api-access-zr2m8:{mountpoint:/var/lib/kubelet/pods/8a707e7f-24ce-4322-b5ad-ffaaa21dc23c/volumes/kubernetes.io~projected/kube-api-access-zr2m8 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93f061fe-5c27-419b-8677-9734d5dce379/volumes/kubernetes.io~projected/kube-api-access-scfnr:{mountpoint:/var/lib/kubelet/pods/93f061fe-5c27-419b-8677-9734d5dce379/volumes/kubernetes.io~projected/kube-api-access-scfnr major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93f061fe-5c27-419b-8677-9734d5dce379/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/93f061fe-5c27-419b-8677-9734d5dce379/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a248c2d8-ba41-4190-9ca7-6cce702dd07a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a248c2d8-ba41-4190-9ca7-6cce702dd07a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a/volumes/kubernetes.io~projected/kube-api-access-pkh5r:{mountpoint:/var/lib/kubelet/pods/a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a/volumes/kubernetes.io~projected/kube-api-access-pkh5r major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf/volumes/kubernetes.io~projected/kube-api-access-c8wtn:{mountpoint:/var/lib/kubelet/pods/a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf/volumes/kubernetes.io~projected/kube-api-access-c8wtn major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2a1e731-e195-4a0b-b778-c606ccc13ef6/volumes/kubernetes.io~projected/kube-api-access-7cwvl:{mountpoint:/var/lib/kubelet/pods/b2a1e731-e195-4a0b-b778-c606ccc13ef6/volumes/kubernetes.io~projected/kube-api-access-7cwvl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3e2fc05-6adb-4c94-a124-aca4483a4e13/volumes/kubernetes.io~projected/kube-api-access-msb4h:{mountpoint:/var/lib/kubelet/pods/b3e2fc05-6adb-4c94-a124-aca4483a4e13/volumes/kubernetes.io~projected/kube-api-access-msb4h major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3e2fc05-6adb-4c94-a124-aca4483a4e13/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b3e2fc05-6adb-4c94-a124-aca4483a4e13/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4376a74-205b-45a1-b005-c59fae4f4201/volumes/kubernetes.io~projected/kube-api-access-wthfq:{mountpoint:/var/lib/kubelet/pods/b4376a74-205b-45a1-b005-c59fae4f4201/volumes/kubernetes.io~projected/kube-api-access-wthfq major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4376a74-205b-45a1-b005-c59fae4f4201/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b4376a74-205b-45a1-b005-c59fae4f4201/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d37e01-bf52-40e0-bc5a-524ce20ddad4/volumes/kubernetes.io~projected/kube-api-access-fzzjh:{mountpoint:/var/lib/kubelet/pods/d8d37e01-bf52-40e0-bc5a-524ce20ddad4/volumes/kubernetes.io~projected/kube-api-access-fzzjh major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30414b5-8768-4eb1-a4ba-395115dba647/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e30414b5-8768-4eb1-a4ba-395115dba647/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30414b5-8768-4eb1-a4ba-395115dba647/volumes/kubernetes.io~projected/kube-api-access-cj8mz:{mountpoint:/var/lib/kubelet/pods/e30414b5-8768-4eb1-a4ba-395115dba647/volumes/kubernetes.io~projected/kube-api-access-cj8mz major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30414b5-8768-4eb1-a4ba-395115dba647/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e30414b5-8768-4eb1-a4ba-395115dba647/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebc0553b-387f-438a-ab73-a053b37abcdd/volumes/kubernetes.io~projected/kube-api-access-2jvvh:{mountpoint:/var/lib/kubelet/pods/ebc0553b-387f-438a-ab73-a053b37abcdd/volumes/kubernetes.io~projected/kube-api-access-2jvvh major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebc0553b-387f-438a-ab73-a053b37abcdd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ebc0553b-387f-438a-ab73-a053b37abcdd/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7aac07e-1e6f-4f62-897c-0061aa870749/volumes/kubernetes.io~projected/kube-api-access-zjmlj:{mountpoint:/var/lib/kubelet/pods/f7aac07e-1e6f-4f62-897c-0061aa870749/volumes/kubernetes.io~projected/kube-api-access-zjmlj major:0 minor:223 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4ef3dfc3b6ba3a8f0372009c8b05a74f3117ab61d7910fff22fa4154a4ac8b97/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/be9eda67e3ec23fa9025d6623a33da87152c7066d52fe0441703fd67d40baccb/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a0929dda07a6fb6d3ed90f4a63b8479b8b74d5025b73b81aca655eac166fefcb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/67c5b16628e44a90f929c671ff668a9f8087377de8b9647bf74058d897a15361/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3ed6c2a030b1bed6c8cfe158da9964d01588472034ff8147f2a7644521d9e722/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7c317932714ccb5be269947e98e1d55ba6f9a9f7d0c2c8b05274f316aa2cbf9c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/82004f1a76fad8a5c31624a03be49a7a90d8ddcabf90e697a1688c12c90d735d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6e6424c41d32995a22eafd34ca04b5d0ab3cb7f8c289674a659a667ae2b819b2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/a5635647f70c76030886e7851cde525baa008a172194dac1edb4b1741d62b23d/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/1005cde677656336f202460994ccb6ae4257e7dd59416ae1091478ef1d6b5a25/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/c25705927403fd97a37e38668f59e561591f8b977aff518e303e65d1108feaae/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/6cd5da69d2aec38d6a958b73f2109a084ed2d51fb19970366818108b99deaece/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/eb43cc0a781b21e6a6ac42651625e3220cc9bf77dda86fc9f7b7f3b4cf33a37d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/5100b957305149eee82206b9855f2ee647429863115566859ce902b812d426c3/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/46fde66b25696b86808b728381bb4eba09814f6ddc63136db8505327fc753bdb/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d21d98a47168ca891d6d20251185ef5c72383f3362f52423e99490ffbc7a6557/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b41bdbbf8698bc28b04085e0b856352a7760846a21449b73434e61208078e901/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/98d484c021732533e1dc6e722b27eeb5fc240e44879610ddbb36701e680c1e47/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/d2a42308ec60504a671c3b93728f38a36a6825d80f79a276c7a6fb9f8f59f4e4/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-183:{mountpoint:/var/lib/containers/storage/overlay/e8b1aebcfeaf8437ba8db37408180483f37f4604bf8d12ee2ade1556fabee1df/merged major:0 minor:183 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/199b59c10da499643f5e44a1867f6c0a7f356ee719cb12a7ca614ca1b837567f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/8404e7c6237285b42364e33e0f12be6ae330433e51ede508e888ee63c6ab9b64/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/3f294ef39937ff3b85dcc3aef274c935d7f7a44f5e015c2f359b38d29b394ed4/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/d31ea02a2b854b1b3f21be2d1a7261e70779f7a20dcd0850d71d3b5b3dd1a7a1/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/d60ac15781622533092b1f750c583bbfa02b5caa4c07f8d3313b12f69f0c0c59/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/0ce01715e45bc05df056029d81bf40c7ad81e276cf9d9a2b4c9329b6a877a79f/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/05784247b0a9d30afa6d1da6bc4df7a3d9443a117666efa56b3f4174703028d4/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/cbdaf533460a3695ddbd9efe7f4d725ca901750791f00c791c56935532a93e37/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/03fba394c34f8a7e7ecebeb6df5b39cddc6da5764b14f6bf028fd7a223dce278/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/15ecf1f72ad030f5025458b568479a0d734b788dffd94f268bbc775366d8a8ff/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/201759bbb4c017f6dfbf104451c616c25730eef4d28616fd2c0aefd1bbf9dcec/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/9f3e4ef438552f82c516ca3dc49b7765f74130d90f352b5f37156160738cff44/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/d7e8ec36b76be98bbb7547f6757966cb8240e8678d1026644256e23711b635db/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/2b97aaabf04104f3b5b1a2c93063c06c595814f66ef98b2a14972616a1a96474/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/7bdf09b7d355c1e7a1a6420a96d9c9b88085e822d979ae73f6241ca9b503ec53/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3862a91e21fb81e6e7146835744e83def14bc7680ef61f28910a55cbabae474f/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/49bfc368e4614b02c40a07074aad28b5f9610035d88e5631b7402eb0aee158a6/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/c6d7291bb57bc957800dceb1f951e9d6ba28e59fbf5f904eb356b2ecb39e7c54/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/1b042f47e5587f151d9a6692e3f9b249db5fe271f040fdefdbd63888ac982cf1/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a30d0e7d6b47d0b86bc1db3b6b7580f09fbe18179b583f2d7dda1d8057da34d3/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/3edb19a0688db206e869cb71b1005c93e90e51baab2195e34a0ba25ed46e37cf/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/0fc3f5fab7d106a50a5f13877998c56c05419f6cd8568caf58361a003e154d83/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/09645ab830b7400c4eef2c9aab63df5274ee358e3068230da7cd38d174349585/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/06e694019a9edc4c20f7a278dd93a555546ccfe7da77ef0dd928071da4f2f792/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/540f72006d4bf965831c031b9f83a4705134c40bcadda3410ab0c9e6c3464072/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/02637aa604e94899191999d55619721e0eed7888e9d754b00204c6dab56558a0/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/1534869fc3f4ad67e57901adaa86e7b26decd48a2361cbb1e3693dbf4910a9b5/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/5e66ab6542264b4cbdb975f6043c98899d8f6d712ca01e62fe0e3599d4d68bde/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/4c9fa153fa96029ebea38ce3f82844ff7e751c565da684013ea0ba454b737c63/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/6ef63756650c4e98203f38f7f57fc97819b4f384bb197f1f098e3dad5222690c/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/dc665cee530040d6f931bfac776d20a2e8fd6e9c03315bdf7fbed257e4a7198d/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/14161f982d1c48c872d2fa595706a8caf27ffaf1d063179217acb4b73abb7245/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/057943f22f18edb5ad120487edad613edea3be2d982ff714dfdd7648f01fea9f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8d49e58b87d9be7311182a1b112f2c332015c9cc79761e078f8e37eec5ac56b4/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a45d5a00e4179856429f6905fb484a59d041ef57e2797603e7da6e501fa59644/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/e1dbb06104e5b116cdd28caf8fbdeece5d58cd5327380885b6e37c7f91081463/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/033d773ff533a126f903bd74a4e26d47a05973d73b21855ddbf35cd7087d0270/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/040949fe0559ba5134820a758da28eeba756f478b47f7535e8d4724d3dab2559/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/2806cd72498bd0379022a9828f5dc5dccc683ce0cc2b21d90ed35519eb9dcee9/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/47b172123789158aa561417ec00eb8c189e0bacbe5f30e77a04e000f310000c3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d2d508746db8065a5724ba0704105c0575d61de308e5705c9c5d4c52140973d5/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/9cfc6358406802a13864e022b0641db136eb85818baf53c6f210cb7128b507ec/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 20:37:13.610177 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.609657    4810 manager.go:217] Machine: {Timestamp:2026-10-30 20:37:13.609161041 +0000 UTC m=+0.100977577 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100126 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e67722f9f68cd563b728fc8088185 SystemUUID:ec2e6772-2f9f-68cd-563b-728fc8088185 BootID:8e42368f-59cc-4a19-9a7b-bf69fc9f7e6d Filesystems:[{Device:/var/lib/kubelet/pods/d8d37e01-bf52-40e0-bc5a-524ce20ddad4/volumes/kubernetes.io~projected/kube-api-access-fzzjh DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d7898c1-9d6c-4382-baa2-e6baae34941a/volumes/kubernetes.io~projected/kube-api-access-b94nz DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e17ab44add68a3afb1672716b073e1a15d617e833b45b102dde94aa0d60f74e7/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b23ef6f859a17d83bde3e36d35d2ae515a76bfea55f5c2762c57472d70669a2/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2a1e731-e195-4a0b-b778-c606ccc13ef6/volumes/kubernetes.io~projected/kube-api-access-7cwvl DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/72c43d73-596a-48d7-aa34-60c720b92fb6/volumes/kubernetes.io~projected/kube-api-access-khdc5 DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e30414b5-8768-4eb1-a4ba-395115dba647/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27c3a01f6e08f36046a0ba85282af82e507a7c028a684a99bdd2e8c672e8a2d7/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e454def1ef9c855b78b8c4a7113721c71551983116c56705ddb8cd79a617fd43/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f0e7657fb4e466651c4da1b3cede4610cc2c7ab47b0c4fc3915253fb14f340d/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b2528cedc5eec4e212563e57e0ff38931265057f3bf59f9d1d47d51a42f2284/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1e310a1152b31ec7ee355d51efa554069bcd8a6fb1c07c58982aa4264b9b55a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/395a48bf-cfce-443e-be13-b2939a9b9b8a/volumes/kubernetes.io~projected/kube-api-access-khjfm DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7fc47c20-4a62-4167-9334-4d7b1cadb5b5/volumes/kubernetes.io~projected/kube-api-access-dhwsm DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4376a74-205b-45a1-b005-c59fae4f4201/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7389168a-7f19-48df-98f8-6d6d20892416/volumes/kubernetes.io~projected/kube-api-access-2bv5x DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ed8f5dbe5f679f34e4c7e968461203bd8f366a76aebfff9e050a3c135e76bc1/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/585e93ee-0fc2-4461-b5ad-e9e0f50afc06/volumes/kubernetes.io~projected/kube-api-access-hqrlf DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d7877a9-9cc5-4655-9b03-24fd447878ac/volumes/kubernetes.io~projected/kube-api-access-k2jfd DeviceMajor:0 DeviceMinor:347 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf/volumes/kubernetes.io~projected/kube-api-access-c8wtn DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89894ec91743e630b725b87d833a6bf3dc8968b83f1f1bfd6b6825f786df8de9/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d9649e08d287f9cf3755bdc1e0fd411c4abdfd438f4612f03ff2217c604f5c2/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f326d3df10a2dc09503dae8b594c653b2397e25df090e25d839dac4162fb7ef9/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8463e4ae87cc0710ca9aad3fcffda20a840c5a06db2a0c113445af48fe880008/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ebc0553b-387f-438a-ab73-a053b37abcdd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0c8d484c3330acc4ec03058786ab4f68a8cd5fbde42e9da92f9bf28e19f3ec6/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93f061fe-5c27-419b-8677-9734d5dce379/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~projected/kube-api-access-v7b7j DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fc7e09d5a272e8b6c1a912933f9ed70ff39777b9df34e476cda8442364a703b/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3e2fc05-6adb-4c94-a124-aca4483a4e13/volumes/kubernetes.io~projected/kube-api-access-msb4h DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a707e7f-24ce-4322-b5ad-ffaaa21dc23c/volumes/kubernetes.io~projected/kube-api-access-zr2m8 DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~projected/kube-api-access-jvtbb DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93f061fe-5c27-419b-8677-9734d5dce379/volumes/kubernetes.io~projected/kube-api-access-scfnr DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e8ae9789e9c92ac14c322e2538f04efc9662b1e1e7fb068b9994c3b513cec0e/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-183 DeviceMajor:0 DeviceMinor:183 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b7e77e725cde65cecf7918764087c2c53584f845186c677033925e14d6db3b7/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ba0647d-e918-4bc7-843b-205e600536ec/volumes/kubernetes.io~projected/kube-api-access-2qgww DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84c4b87f449036eaa626d4e1149ae5e7c367127b152e827259d9f91129c5a168/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ba0647d-e918-4bc7-843b-205e600536ec/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3efcea4c-b6b6-47a5-be1d-c61a83bbe907/volumes/kubernetes.io~projected/kube-api-access-qnxvc DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d4395f4806e92dfb0cbfee2babccbd31159975836f2db0df3a1548716eb312f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dad5510556bc61a7a7c6d5b3110720bcfe7b95fc35c5038fa78e3c1d72256a4/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/73fa532a-7546-4e60-8f8c-901cebfd7f07/volumes/kubernetes.io~projected/kube-api-access-7zlkc DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e30414b5-8768-4eb1-a4ba-395115dba647/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a/volumes/kubernetes.io~projected/kube-api-access-pkh5r DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4eccb371c44a504ecceba60e71d99b241c6719656d1e3153e55d3e4c58cdc01f/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e30414b5-8768-4eb1-a4ba-395115dba647/volumes/kubernetes.io~projected/kube-api-access-cj8mz DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3e2fc05-6adb-4c94-a124-aca4483a4e13/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b53bf2d99bd9bb10291cc1130c05774b10c78c543f5a74db35199f5b662b06d5/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2fc2fd6c-3d77-4446-9402-cf73cdd3497a/volumes/kubernetes.io~projected/kube-api-access-m8ksf DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a/volumes/kubernetes.io~projected/kube-api-access-wxfsd DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4376a74-205b-45a1-b005-c59fae4f4201/volumes/kubernetes.io~projected/kube-api-access-wthfq DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6140cde0-b52b-48e1-a1d6-bcac35086922/volumes/kubernetes.io~projected/kube-api-access-wmcl5 DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd6a2570855240aa73ed8dc4ce409b82a7a0eba2fc5762c15c3c61341c0921be/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51d19e04-9e59-4f4f-be5c-9900cca6d655/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58b187ea-dead-4a37-8a52-5e447783d815/volumes/kubernetes.io~projected/kube-api-access-gpbbp DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d2edfa6-676f-47ab-af10-37be9080c821/volumes/kubernetes.io~projected/kube-api-access-xqcxr DeviceMajor:0 DeviceMinor:342 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/volumes/kubernetes.io~projected/kube-api-access-plcbb DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a248c2d8-ba41-4190-9ca7-6cce702dd07a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:274 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/47697b9d-ed21-4e3b-93b0-320138d4037d/volumes/kubernetes.io~projected/kube-api-access-dtw88 DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d7877a9-9cc5-4655-9b03-24fd447878ac/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:346 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ba0647d-e918-4bc7-843b-205e600536ec/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/395a48bf-cfce-443e-be13-b2939a9b9b8a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ebc0553b-387f-438a-ab73-a053b37abcdd/volumes/kubernetes.io~projected/kube-api-access-2jvvh DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/591c79834fe538885aa02a531b5fa1849744a49dda1c869cca457167cd429429/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3a0c9d437acf8b3dae52da884f0c764c3f8f8b01bf4947dd9a149c635bc1584/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/450407d6c8501801d763d37fa3ef48d218689f995b03dc5ed8bc4fce95a3e554/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5f43122589f19a195e82f24618908f0f05878856b1a3e1d4bb2f18609ecfc7d/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8902c3bd-bf64-48e1-b113-d42ed8456348/volumes/kubernetes.io~projected/kube-api-access-xph2w DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7aac07e-1e6f-4f62-897c-0061aa870749/volumes/kubernetes.io~projected/kube-api-access-zjmlj DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0fc7e09d5a272e8 MacAddress:f6:6f:56:48:99:4e Speed:10000 Mtu:8901} {Name:1d4395f4806e92d MacAddress:c2:9b:90:e6:b7:c4 Speed:10000 Mtu:8901} {Name:1f0e7657fb4e466 MacAddress:7e:f5:50:2a:af:f6 Speed:10000 Mtu:8901} {Name:4eccb371c44a504 MacAddress:7e:45:13:9f:ff:41 Speed:10000 Mtu:8901} {Name:591c79834fe5388 MacAddress:0e:c7:79:8b:55:62 Speed:10000 Mtu:8901} {Name:5b7e77e725cde65 MacAddress:6a:c9:07:82:5d:43 Speed:10000 Mtu:8901} {Name:6b23ef6f859a17d MacAddress:4a:5c:d9:de:9a:75 Speed:10000 Mtu:8901} {Name:6b2528cedc5eec4 MacAddress:9a:90:f0:6c:bf:5a Speed:10000 Mtu:8901} {Name:6ed8f5dbe5f679f MacAddress:36:d8:30:84:09:49 Speed:10000 Mtu:8901} {Name:84c4b87f449036e MacAddress:f2:21:4e:20:ff:ab Speed:10000 Mtu:8901} {Name:89894ec91743e63 MacAddress:1e:44:5d:5b:2b:fc Speed:10000 Mtu:8901} {Name:8e8ae9789e9c92a MacAddress:46:f6:20:fe:71:27 Speed:10000 Mtu:8901} {Name:b3a0c9d437acf8b MacAddress:c2:32:41:ab:2b:cf Speed:10000 Mtu:8901} {Name:b53bf2d99bd9bb1 MacAddress:b6:22:5f:91:72:54 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e2:c7:ec:cd Speed:0 Mtu:9001} {Name:br-int MacAddress:be:55:ef:48:90:16 Speed:0 Mtu:8901} {Name:e17ab44add68a3a MacAddress:9a:e2:08:44:40:62 Speed:10000 Mtu:8901} {Name:e454def1ef9c855 MacAddress:c2:e4:6a:2a:de:ee Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e2:c7:ec:cd Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:d6:2a:72:bb:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:37:13.610458 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.610184    4810 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:37:13.610458 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.610293    4810 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:37:13.610562 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.610541    4810 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:37:13.610679 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.610562    4810 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-161.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:37:13.610726 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.610682    4810 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:37:13.610726 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.610689    4810 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:37:13.610726 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.610699    4810 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:37:13.610726 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.610718    4810 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:37:13.610829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.610783    4810 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:37:13.610966 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.610956    4810 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:37:13.611074 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.611065    4810 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:37:13.611107 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.611100    4810 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:37:13.611145 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.611137    4810 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:37:13.611179 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.611151    4810 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:37:13.611179 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.611160    4810 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:37:13.611263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.611184    4810 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:37:13.611937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.611922    4810 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:37:13.611986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.611938    4810 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:37:13.612387 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.612370    4810 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:37:13.614739 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.614725    4810 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:37:13.614739 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.614737    4810 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:37:13.614846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.614836    4810 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 20:37:13.615102 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615090    4810 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:37:13.615144 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615120    4810 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:37:13.615224 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615214    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:37:13.615252 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615231    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:37:13.615252 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615240    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:37:13.615252 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615248    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:37:13.615334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615256    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:37:13.615334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615265    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:37:13.615334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615274    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:37:13.615334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615281    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:37:13.615334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615292    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:37:13.615334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615303    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:37:13.615334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615320    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:37:13.615334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615334    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:37:13.615535 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615360    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:37:13.615535 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.615365    4810 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:37:13.619709 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.619694    4810 server.go:1263] "Started kubelet"
Oct 30 20:37:13.619803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.619753    4810 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:37:13.619891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.619827    4810 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:37:13.619959 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.619917    4810 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:37:13.620317 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.620299    4810 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:37:13.620487 ip-10-0-128-161 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:37:13.620961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.620943    4810 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:37:13.623582 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.623566    4810 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:37:13.623582 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.623574    4810 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:37:13.623801 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.623608    4810 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:31:34 +0000 UTC" deadline="2028-06-30 08:22:23.64539231 +0000 UTC"
Oct 30 20:37:13.623801 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.623637    4810 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14603h45m10.021762311s"
Oct 30 20:37:13.623801 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.623684    4810 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:37:13.623801 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.623695    4810 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:37:13.623801 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:13.623735    4810 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-161.us-east-2.compute.internal\" not found"
Oct 30 20:37:13.623801 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.623795    4810 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:37:13.627088 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627056    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2" volumeName="kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-default-certificate" seLinuxMountContext=""
Oct 30 20:37:13.627185 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627092    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fc47c20-4a62-4167-9334-4d7b1cadb5b5" volumeName="kubernetes.io/projected/7fc47c20-4a62-4167-9334-4d7b1cadb5b5-kube-api-access-dhwsm" seLinuxMountContext=""
Oct 30 20:37:13.627185 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627105    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2a1e731-e195-4a0b-b778-c606ccc13ef6" volumeName="kubernetes.io/configmap/b2a1e731-e195-4a0b-b778-c606ccc13ef6-iptables-alerter-script" seLinuxMountContext=""
Oct 30 20:37:13.627185 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627125    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3e2fc05-6adb-4c94-a124-aca4483a4e13" volumeName="kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:37:13.627185 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627140    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d7877a9-9cc5-4655-9b03-24fd447878ac" volumeName="kubernetes.io/secret/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-key" seLinuxMountContext=""
Oct 30 20:37:13.627185 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627152    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395a48bf-cfce-443e-be13-b2939a9b9b8a" volumeName="kubernetes.io/secret/395a48bf-cfce-443e-be13-b2939a9b9b8a-serving-cert" seLinuxMountContext=""
Oct 30 20:37:13.627185 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627176    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf" volumeName="kubernetes.io/projected/a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf-kube-api-access-c8wtn" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627198    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3e2fc05-6adb-4c94-a124-aca4483a4e13" volumeName="kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-tmp" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627221    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebc0553b-387f-438a-ab73-a053b37abcdd" volumeName="kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-config" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627245    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204ba748-a514-4772-9fba-dfafd3eddcf7" volumeName="kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-bound-sa-token" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627262    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd510fb-f40e-401c-845f-23b9d750dc68" volumeName="kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-bound-sa-token" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627278    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2" volumeName="kubernetes.io/projected/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-kube-api-access-plcbb" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627289    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93f061fe-5c27-419b-8677-9734d5dce379" volumeName="kubernetes.io/secret/93f061fe-5c27-419b-8677-9734d5dce379-cert" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627304    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a" volumeName="kubernetes.io/projected/a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a-kube-api-access-pkh5r" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627319    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebc0553b-387f-438a-ab73-a053b37abcdd" volumeName="kubernetes.io/secret/ebc0553b-387f-438a-ab73-a053b37abcdd-serving-cert" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627330    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d7877a9-9cc5-4655-9b03-24fd447878ac" volumeName="kubernetes.io/projected/1d7877a9-9cc5-4655-9b03-24fd447878ac-kube-api-access-k2jfd" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627347    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72c43d73-596a-48d7-aa34-60c720b92fb6" volumeName="kubernetes.io/projected/72c43d73-596a-48d7-aa34-60c720b92fb6-kube-api-access-khdc5" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627365    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3e2fc05-6adb-4c94-a124-aca4483a4e13" volumeName="kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-service-ca-bundle" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627376    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30414b5-8768-4eb1-a4ba-395115dba647" volumeName="kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-config" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627393    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204ba748-a514-4772-9fba-dfafd3eddcf7" volumeName="kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-installation-pull-secrets" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627403    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a707e7f-24ce-4322-b5ad-ffaaa21dc23c" volumeName="kubernetes.io/projected/8a707e7f-24ce-4322-b5ad-ffaaa21dc23c-kube-api-access-zr2m8" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627413    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebc0553b-387f-438a-ab73-a053b37abcdd" volumeName="kubernetes.io/projected/ebc0553b-387f-438a-ab73-a053b37abcdd-kube-api-access-2jvvh" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627428    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d7898c1-9d6c-4382-baa2-e6baae34941a" volumeName="kubernetes.io/projected/0d7898c1-9d6c-4382-baa2-e6baae34941a-kube-api-access-b94nz" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627438    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ba0647d-e918-4bc7-843b-205e600536ec" volumeName="kubernetes.io/projected/2ba0647d-e918-4bc7-843b-205e600536ec-kube-api-access-2qgww" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627448    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd510fb-f40e-401c-845f-23b9d750dc68" volumeName="kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-trusted-ca" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627466    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58b187ea-dead-4a37-8a52-5e447783d815" volumeName="kubernetes.io/projected/58b187ea-dead-4a37-8a52-5e447783d815-kube-api-access-gpbbp" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627476    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="734323fa-6946-426b-9e06-396cb4e7b48f" volumeName="kubernetes.io/configmap/734323fa-6946-426b-9e06-396cb4e7b48f-nginx-conf" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627486    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d37e01-bf52-40e0-bc5a-524ce20ddad4" volumeName="kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627504    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a" volumeName="kubernetes.io/projected/2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a-kube-api-access-wxfsd" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627519    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3e2fc05-6adb-4c94-a124-aca4483a4e13" volumeName="kubernetes.io/secret/b3e2fc05-6adb-4c94-a124-aca4483a4e13-serving-cert" seLinuxMountContext=""
Oct 30 20:37:13.627496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627529    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30414b5-8768-4eb1-a4ba-395115dba647" volumeName="kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-env-overrides" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627547    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ba0647d-e918-4bc7-843b-205e600536ec" volumeName="kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-tmp" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627554    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd510fb-f40e-401c-845f-23b9d750dc68" volumeName="kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627571    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58b187ea-dead-4a37-8a52-5e447783d815" volumeName="kubernetes.io/configmap/58b187ea-dead-4a37-8a52-5e447783d815-serviceca" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627595    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6140cde0-b52b-48e1-a1d6-bcac35086922" volumeName="kubernetes.io/configmap/6140cde0-b52b-48e1-a1d6-bcac35086922-config-volume" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627651    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2" volumeName="kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-stats-auth" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627716    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7389168a-7f19-48df-98f8-6d6d20892416" volumeName="kubernetes.io/projected/7389168a-7f19-48df-98f8-6d6d20892416-kube-api-access-2bv5x" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627751    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3e2fc05-6adb-4c94-a124-aca4483a4e13" volumeName="kubernetes.io/projected/b3e2fc05-6adb-4c94-a124-aca4483a4e13-kube-api-access-msb4h" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627770    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30414b5-8768-4eb1-a4ba-395115dba647" volumeName="kubernetes.io/projected/e30414b5-8768-4eb1-a4ba-395115dba647-kube-api-access-cj8mz" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627808    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47697b9d-ed21-4e3b-93b0-320138d4037d" volumeName="kubernetes.io/configmap/47697b9d-ed21-4e3b-93b0-320138d4037d-telemetry-config" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627872    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2edfa6-676f-47ab-af10-37be9080c821" volumeName="kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-service-ca-bundle" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627899    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2edfa6-676f-47ab-af10-37be9080c821" volumeName="kubernetes.io/empty-dir/2d2edfa6-676f-47ab-af10-37be9080c821-archives-path" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627910    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8902c3bd-bf64-48e1-b113-d42ed8456348" volumeName="kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627942    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2a1e731-e195-4a0b-b778-c606ccc13ef6" volumeName="kubernetes.io/projected/b2a1e731-e195-4a0b-b778-c606ccc13ef6-kube-api-access-7cwvl" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627954    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7aac07e-1e6f-4f62-897c-0061aa870749" volumeName="kubernetes.io/configmap/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-config" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627965    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7aac07e-1e6f-4f62-897c-0061aa870749" volumeName="kubernetes.io/projected/f7aac07e-1e6f-4f62-897c-0061aa870749-kube-api-access-zjmlj" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.627992    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204ba748-a514-4772-9fba-dfafd3eddcf7" volumeName="kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628010    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efcea4c-b6b6-47a5-be1d-c61a83bbe907" volumeName="kubernetes.io/projected/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-kube-api-access-qnxvc" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628024    4810 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628020    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8902c3bd-bf64-48e1-b113-d42ed8456348" volumeName="kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628059    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93f061fe-5c27-419b-8677-9734d5dce379" volumeName="kubernetes.io/projected/93f061fe-5c27-419b-8677-9734d5dce379-kube-api-access-scfnr" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628072    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73fa532a-7546-4e60-8f8c-901cebfd7f07" volumeName="kubernetes.io/projected/73fa532a-7546-4e60-8f8c-901cebfd7f07-kube-api-access-7zlkc" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628080    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd510fb-f40e-401c-845f-23b9d750dc68" volumeName="kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-kube-api-access-jvtbb" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628088    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd510fb-f40e-401c-845f-23b9d750dc68" volumeName="kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-installation-pull-secrets" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628096    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585e93ee-0fc2-4461-b5ad-e9e0f50afc06" volumeName="kubernetes.io/empty-dir/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-data-volume" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628103    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4376a74-205b-45a1-b005-c59fae4f4201" volumeName="kubernetes.io/configmap/b4376a74-205b-45a1-b005-c59fae4f4201-config" seLinuxMountContext=""
Oct 30 20:37:13.628464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628115    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4376a74-205b-45a1-b005-c59fae4f4201" volumeName="kubernetes.io/projected/b4376a74-205b-45a1-b005-c59fae4f4201-kube-api-access-wthfq" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628140    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4376a74-205b-45a1-b005-c59fae4f4201" volumeName="kubernetes.io/secret/b4376a74-205b-45a1-b005-c59fae4f4201-serving-cert" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.628997    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6140cde0-b52b-48e1-a1d6-bcac35086922" volumeName="kubernetes.io/projected/6140cde0-b52b-48e1-a1d6-bcac35086922-kube-api-access-wmcl5" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629017    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8902c3bd-bf64-48e1-b113-d42ed8456348" volumeName="kubernetes.io/projected/8902c3bd-bf64-48e1-b113-d42ed8456348-kube-api-access-xph2w" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629042    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d37e01-bf52-40e0-bc5a-524ce20ddad4" volumeName="kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-daemon-config" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629057    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47697b9d-ed21-4e3b-93b0-320138d4037d" volumeName="kubernetes.io/projected/47697b9d-ed21-4e3b-93b0-320138d4037d-kube-api-access-dtw88" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629078    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd510fb-f40e-401c-845f-23b9d750dc68" volumeName="kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-certificates" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629093    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204ba748-a514-4772-9fba-dfafd3eddcf7" volumeName="kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-certificates" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629236    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2edfa6-676f-47ab-af10-37be9080c821" volumeName="kubernetes.io/projected/2d2edfa6-676f-47ab-af10-37be9080c821-kube-api-access-xqcxr" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629254    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395a48bf-cfce-443e-be13-b2939a9b9b8a" volumeName="kubernetes.io/projected/395a48bf-cfce-443e-be13-b2939a9b9b8a-kube-api-access-khjfm" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629368    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585e93ee-0fc2-4461-b5ad-e9e0f50afc06" volumeName="kubernetes.io/projected/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-api-access-hqrlf" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629389    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30414b5-8768-4eb1-a4ba-395115dba647" volumeName="kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629431    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204ba748-a514-4772-9fba-dfafd3eddcf7" volumeName="kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-trusted-ca" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629445    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204ba748-a514-4772-9fba-dfafd3eddcf7" volumeName="kubernetes.io/empty-dir/204ba748-a514-4772-9fba-dfafd3eddcf7-ca-trust-extracted" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629487    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d2edfa6-676f-47ab-af10-37be9080c821" volumeName="kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629500    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8902c3bd-bf64-48e1-b113-d42ed8456348" volumeName="kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629605    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3e2fc05-6adb-4c94-a124-aca4483a4e13" volumeName="kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-snapshots" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629618    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebc0553b-387f-438a-ab73-a053b37abcdd" volumeName="kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-trusted-ca" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629634    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7389168a-7f19-48df-98f8-6d6d20892416" volumeName="kubernetes.io/empty-dir/7389168a-7f19-48df-98f8-6d6d20892416-tmp-dir" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629653    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d7877a9-9cc5-4655-9b03-24fd447878ac" volumeName="kubernetes.io/configmap/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-cabundle" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629666    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204ba748-a514-4772-9fba-dfafd3eddcf7" volumeName="kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-kube-api-access-v7b7j" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629682    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fc2fd6c-3d77-4446-9402-cf73cdd3497a" volumeName="kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629695    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd510fb-f40e-401c-845f-23b9d750dc68" volumeName="kubernetes.io/empty-dir/4dd510fb-f40e-401c-845f-23b9d750dc68-ca-trust-extracted" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629708    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6140cde0-b52b-48e1-a1d6-bcac35086922" volumeName="kubernetes.io/empty-dir/6140cde0-b52b-48e1-a1d6-bcac35086922-tmp-dir" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629740    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a248c2d8-ba41-4190-9ca7-6cce702dd07a" volumeName="kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629752    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d37e01-bf52-40e0-bc5a-524ce20ddad4" volumeName="kubernetes.io/projected/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-kube-api-access-fzzjh" seLinuxMountContext=""
Oct 30 20:37:13.629829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629784    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30414b5-8768-4eb1-a4ba-395115dba647" volumeName="kubernetes.io/secret/e30414b5-8768-4eb1-a4ba-395115dba647-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 20:37:13.631045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629796    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ba0647d-e918-4bc7-843b-205e600536ec" volumeName="kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-etc-tuned" seLinuxMountContext=""
Oct 30 20:37:13.631045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629924    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395a48bf-cfce-443e-be13-b2939a9b9b8a" volumeName="kubernetes.io/configmap/395a48bf-cfce-443e-be13-b2939a9b9b8a-config" seLinuxMountContext=""
Oct 30 20:37:13.631045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629951    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51d19e04-9e59-4f4f-be5c-9900cca6d655" volumeName="kubernetes.io/configmap/51d19e04-9e59-4f4f-be5c-9900cca6d655-konnectivity-ca" seLinuxMountContext=""
Oct 30 20:37:13.631045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629964    4810 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51d19e04-9e59-4f4f-be5c-9900cca6d655" volumeName="kubernetes.io/secret/51d19e04-9e59-4f4f-be5c-9900cca6d655-agent-certs" seLinuxMountContext=""
Oct 30 20:37:13.631045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629975    4810 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:37:13.631045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629983    4810 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:37:13.631045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.629992    4810 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:13.631732 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.631711    4810 factory.go:55] Registering systemd factory
Oct 30 20:37:13.631732 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.631731    4810 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:37:13.632305 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.632286    4810 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:13.632873 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.632845    4810 factory.go:153] Registering CRI-O factory
Oct 30 20:37:13.632873 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.632873    4810 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:37:13.632951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.632924    4810 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:37:13.632951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.632943    4810 factory.go:103] Registering Raw factory
Oct 30 20:37:13.633004 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.632954    4810 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:37:13.633214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.633205    4810 manager.go:319] Starting recovery of all containers
Oct 30 20:37:13.634529 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.634486    4810 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 20:37:13.645463 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.645353    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/1.log"
Oct 30 20:37:13.646143 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.646124    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/1.log"
Oct 30 20:37:13.652056 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.652003    4810 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:37:13.653840 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.653622    4810 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:37:13.653840 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.653661    4810 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:37:13.653840 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.653683    4810 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:37:13.653840 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:13.653742    4810 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 20:37:13.658007 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.657990    4810 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:13.670091 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.670071    4810 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="ca09d1bca6b4ef19b9ebfc2e4f14015c4fcb6da72462c465b8650669ae8907e1" exitCode=0
Oct 30 20:37:13.670091 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.670089    4810 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="dd6da7b9b0c6b3c90ae47d10799617ef6acc1dc11708d35296dd689550a2655e" exitCode=0
Oct 30 20:37:13.670200 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.670096    4810 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="bf64e111bc2eb938f9a13944a76f6a793a425f99148ab72b39ea6179ae73b26c" exitCode=0
Oct 30 20:37:13.670200 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.670102    4810 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="5d53003e0cf235e30f997c835c6344e1038e533f7abb650d4804ca7471068e54" exitCode=0
Oct 30 20:37:13.670200 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.670107    4810 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="dd9a7a856055be0b7ad43dcb8223c7222afe1ad7bce172d72f8450159414e9e7" exitCode=0
Oct 30 20:37:13.670200 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.670112    4810 generic.go:356] "Generic (PLEG): container finished" podID="8902c3bd-bf64-48e1-b113-d42ed8456348" containerID="9c4777506e01804a8ad316b1f2133c5d65ad9cb5230d922abbb3aef9d7b9ce73" exitCode=0
Oct 30 20:37:13.671731 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.671716    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/1.log"
Oct 30 20:37:13.671809 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.671741    4810 generic.go:356] "Generic (PLEG): container finished" podID="ebc0553b-387f-438a-ab73-a053b37abcdd" containerID="f354ab5f6dc4cbe1be1a150d8a98d993f7b1d3c8cf4857d96e6ad15eecfa725f" exitCode=255
Oct 30 20:37:13.674825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.674807    4810 generic.go:356] "Generic (PLEG): container finished" podID="e2241538db95cc825a75346da6583a23" containerID="c117e6fa0635cae5410bf81eda29098b73d71d894b45bfa987370e8056af0879" exitCode=0
Oct 30 20:37:13.728666 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.728612    4810 manager.go:324] Recovery completed
Oct 30 20:37:13.748667 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.748644    4810 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:37:13.748667 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.748659    4810 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:37:13.748846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.748684    4810 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:37:13.748846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.748827    4810 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 20:37:13.748846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.748837    4810 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 20:37:13.748981 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.748852    4810 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 20:37:13.748981 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.748872    4810 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 20:37:13.748981 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.748877    4810 policy_none.go:50] "Start"
Oct 30 20:37:13.748981 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.748886    4810 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:37:13.748981 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.748895    4810 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:37:13.749165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.749040    4810 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 20:37:13.749165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.749047    4810 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 20:37:13.749165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.749052    4810 policy_none.go:44] "Start"
Oct 30 20:37:13.753843 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:13.753823    4810 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 20:37:13.753941 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.753899    4810 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:37:13.756516 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.756500    4810 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:37:13.756972 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:13.756524    4810 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:37:13.756972 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.756971    4810 server.go:87] "Starting device plugin registration server"
Oct 30 20:37:13.757215 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.757202    4810 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:37:13.757248 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.757217    4810 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:37:13.757340 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.757317    4810 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:37:13.757493 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.757458    4810 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:37:13.757559 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.757498    4810 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:37:13.757559 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.757504    4810 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:37:13.757653 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.757545    4810 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:37:13.75748862Z","UUID":"830ad7fa-7eb4-4c19-b7ea-56d76ffd0900","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:37:13.759269 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:13.759247    4810 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:37:13.760004 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.759988    4810 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:37:13.760118 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.760012    4810 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:37:13.858215 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.858181    4810 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:37:13.859784 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.859762    4810 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:37:13.859915 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.859798    4810 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:37:13.859915 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.859813    4810 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-161.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:37:13.859915 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.859883    4810 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:13.872095 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.872074    4810 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:13.872172 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.872135    4810 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:13.954142 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.953945    4810 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"]
Oct 30 20:37:13.954753 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.954710    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal" event={"ID":"157678277e297843d51e22f1021ace90","Type":"ContainerStarted","Data":"1f5a051a5f80989fbea5ad8634938a19b153873f0145352bc86d7efa771f526a"}
Oct 30 20:37:13.954753 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.954749    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal" event={"ID":"157678277e297843d51e22f1021ace90","Type":"ContainerStarted","Data":"fd6a2570855240aa73ed8dc4ce409b82a7a0eba2fc5762c15c3c61341c0921be"}
Oct 30 20:37:13.954936 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.954761    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal" event={"ID":"e2241538db95cc825a75346da6583a23","Type":"ContainerStarted","Data":"a1b4a59f50b719362f249dbbfe1fd01733c5884406e210b5899175bd88263257"}
Oct 30 20:37:13.954936 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.954775    4810 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:13.954936 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.954914    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal" event={"ID":"e2241538db95cc825a75346da6583a23","Type":"ContainerDied","Data":"c117e6fa0635cae5410bf81eda29098b73d71d894b45bfa987370e8056af0879"}
Oct 30 20:37:13.955069 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.954946    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal" event={"ID":"e2241538db95cc825a75346da6583a23","Type":"ContainerStarted","Data":"4d9649e08d287f9cf3755bdc1e0fd411c4abdfd438f4612f03ff2217c604f5c2"}
Oct 30 20:37:13.955069 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.955029    4810 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:13.971466 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.971438    4810 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:37:13.971556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:13.971477    4810 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:37:13.971556 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:13.971504    4810 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:13.971669 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:13.971573    4810 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.036976 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.036921    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.036976 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.036952    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.036976 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.036970    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/157678277e297843d51e22f1021ace90-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"157678277e297843d51e22f1021ace90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.137965 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.137937    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.137965 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.137968    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.138148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.137988    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/157678277e297843d51e22f1021ace90-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"157678277e297843d51e22f1021ace90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.138148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.138038    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/157678277e297843d51e22f1021ace90-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"157678277e297843d51e22f1021ace90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.138148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.138051    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.138148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.138056    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2241538db95cc825a75346da6583a23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" (UID: \"e2241538db95cc825a75346da6583a23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.611964 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.611933    4810 apiserver.go:52] "Watching apiserver"
Oct 30 20:37:14.621123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.621101    4810 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:37:14.621761 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.621740    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wt7nr","openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd","openshift-insights/insights-operator-7c4986d449-rbf5z","openshift-image-registry/image-registry-948b95b59-tdzdl","openshift-cluster-node-tuning-operator/tuned-bkx7g","openshift-dns/dns-default-nfvzn","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n","kube-system/konnectivity-agent-vgcxk","openshift-ingress-canary/ingress-canary-x9v7g","openshift-image-registry/image-registry-5bfdcc7f98-zlptz","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns","openshift-insights/periodic-gathering-7k6rz-jhhmx","openshift-network-diagnostics/network-check-target-lfl79","openshift-ingress/router-default-7dc99f8759-pzknq","openshift-multus/multus-clqzg","kube-system/kas-connection-checker-685895c7c9-jr5cw","kube-system/kas-connection-checker-685895c7c9-nbh76","openshift-network-operator/iptables-alerter-nm42c","openshift-service-ca/service-ca-5fffc97bdf-zqt6s","openshift-console-operator/console-operator-7cd56b8dbc-8nltv","kube-system/kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq","kube-system/global-pull-secret-syncer-vmcq2","openshift-image-registry/node-ca-sq5lx","openshift-dns/node-resolver-4mv46","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs","openshift-multus/multus-additional-cni-plugins-88xhl","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg","openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf","openshift-multus/network-metrics-daemon-gtkw8","openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b","kube-system/kas-connection-checker-685895c7c9-sp5fq","openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2","openshift-ovn-kubernetes/ovnkube-node-k4spg","openshift-insights/insights-runtime-extractor-kz7s8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"]
Oct 30 20:37:14.622939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.622915    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:14.623282 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.623247    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:14.624834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.624816    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.626064 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.626039    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.626160 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.626049    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.626979 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.626956    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.627534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.627510    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.627647 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.627633    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.627942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.627923    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 20:37:14.628148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.628132    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 20:37:14.628231 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.628205    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cg4v8\""
Oct 30 20:37:14.628675 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.628650    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-87d8c\""
Oct 30 20:37:14.629197 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.629174    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 20:37:14.629197 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.629192    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 20:37:14.629393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.629376    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.629616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.629601    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.629785 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.629770    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 20:37:14.630195 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.630177    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 20:37:14.630290 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.630266    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5k9jw\""
Oct 30 20:37:14.630525 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.630511    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:14.630968 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.630948    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sdwp2\""
Oct 30 20:37:14.631110 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.631095    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-btxn9\""
Oct 30 20:37:14.631200 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.631183    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bpj8f\""
Oct 30 20:37:14.631270 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.631249    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 20:37:14.631317 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.631150    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 20:37:14.631317 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.631187    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 20:37:14.631552 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.631534    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 20:37:14.631672 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.631600    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-srgh4\""
Oct 30 20:37:14.632057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632032    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 20:37:14.632173 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632097    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.632528 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632503    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.632673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632551    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.632673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632562    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.632673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632570    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.632673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632585    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.632673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632611    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 20:37:14.632673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632632    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.632673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632635    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 20:37:14.632673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632656    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.632918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632775    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.632918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632811    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.632918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.632881    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:37:14.633299 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.633277    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:37:14.633763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.633747    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-278b6\""
Oct 30 20:37:14.635067 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635045    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.635147 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635124    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 20:37:14.635147 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635133    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 20:37:14.635262 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635188    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 20:37:14.635262 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635184    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.635262 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635225    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6fg9b\""
Oct 30 20:37:14.635382 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635262    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 20:37:14.635382 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635264    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.635382 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635299    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2zkwq\""
Oct 30 20:37:14.635522 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635450    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 20:37:14.635522 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635450    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 20:37:14.635586 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.635576    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.637434 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.637416    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.638838 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.638810    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 20:37:14.639009 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.638988    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:37:14.639514 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.639493    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:14.640092 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.640069    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:37:14.640216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.640183    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 20:37:14.640399 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.640380    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 20:37:14.640399 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.640393    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.640606 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.640576    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.640686 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.640612    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 20:37:14.640786 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.640770    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.640835 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.640612    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.640938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.640916    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.641052 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.640960    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:37:14.641052 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.641015    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:37:14.641154 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.641139    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.641296 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.641283    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.641528 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.641510    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.641616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.641593    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.641737 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.641677    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-82bc7\""
Oct 30 20:37:14.641798 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.641785    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:37:14.642138 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642115    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:37:14.642230 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642188    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x97dd\""
Oct 30 20:37:14.642335 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642314    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4pvnz\""
Oct 30 20:37:14.642391 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642356    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 20:37:14.642462 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642448    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:37:14.642513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642502    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-sbwq8\""
Oct 30 20:37:14.642612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642594    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.642668 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642654    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-chh5h\""
Oct 30 20:37:14.642668 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642118    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fxgqk\""
Oct 30 20:37:14.642783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642766    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:37:14.642942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642924    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bcgjk\""
Oct 30 20:37:14.643011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642995    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jrl7k\""
Oct 30 20:37:14.643165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643140    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:37:14.643341 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643308    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.643434 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643367    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2hx48\""
Oct 30 20:37:14.643434 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.642933    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:37:14.643434 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643398    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rsx8b\""
Oct 30 20:37:14.643576 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643441    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:37:14.643576 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643467    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zljsp\""
Oct 30 20:37:14.643576 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643314    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.643721 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643611    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l4rxj\""
Oct 30 20:37:14.643721 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643667    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:37:14.643721 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643671    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.643850 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643735    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:37:14.643850 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643766    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7lq4m\""
Oct 30 20:37:14.643959 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643909    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:14.644020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643961    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-trusted-ca\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:14.644020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.643967    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.644159 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.644052    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8wtn\" (UniqueName: \"kubernetes.io/projected/a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf-kube-api-access-c8wtn\") pod \"network-check-source-6f78d59f99-fhbw2\" (UID: \"a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2"
Oct 30 20:37:14.644264 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.644247    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.644341 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.644252    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:37:14.644394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.644361    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:37:14.644495 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.644185    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:37:14.644584 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.644521    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkh5r\" (UniqueName: \"kubernetes.io/projected/a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a-kube-api-access-pkh5r\") pod \"kas-connection-checker-685895c7c9-nbh76\" (UID: \"a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a\") " pod="kube-system/kas-connection-checker-685895c7c9-nbh76"
Oct 30 20:37:14.644636 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.644329    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.644672 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.644655    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/47697b9d-ed21-4e3b-93b0-320138d4037d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:14.644708 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.644686    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-config\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:14.644749 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.644707    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/395a48bf-cfce-443e-be13-b2939a9b9b8a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:14.645024 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645008    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwsm\" (UniqueName: \"kubernetes.io/projected/7fc47c20-4a62-4167-9334-4d7b1cadb5b5-kube-api-access-dhwsm\") pod \"kas-connection-checker-685895c7c9-sp5fq\" (UID: \"7fc47c20-4a62-4167-9334-4d7b1cadb5b5\") " pod="kube-system/kas-connection-checker-685895c7c9-sp5fq"
Oct 30 20:37:14.645065 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645044    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebc0553b-387f-438a-ab73-a053b37abcdd-serving-cert\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:14.645102 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645068    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvvh\" (UniqueName: \"kubernetes.io/projected/ebc0553b-387f-438a-ab73-a053b37abcdd-kube-api-access-2jvvh\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:14.645102 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645086    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/395a48bf-cfce-443e-be13-b2939a9b9b8a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:14.645182 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645100    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khjfm\" (UniqueName: \"kubernetes.io/projected/395a48bf-cfce-443e-be13-b2939a9b9b8a-kube-api-access-khjfm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:14.645182 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645116    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:14.645182 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645132    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjmlj\" (UniqueName: \"kubernetes.io/projected/f7aac07e-1e6f-4f62-897c-0061aa870749-kube-api-access-zjmlj\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:14.645182 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645155    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:14.645182 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645172    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtw88\" (UniqueName: \"kubernetes.io/projected/47697b9d-ed21-4e3b-93b0-320138d4037d-kube-api-access-dtw88\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:14.645395 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645194    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2m8\" (UniqueName: \"kubernetes.io/projected/8a707e7f-24ce-4322-b5ad-ffaaa21dc23c-kube-api-access-zr2m8\") pod \"kas-connection-checker-685895c7c9-jr5cw\" (UID: \"8a707e7f-24ce-4322-b5ad-ffaaa21dc23c\") " pod="kube-system/kas-connection-checker-685895c7c9-jr5cw"
Oct 30 20:37:14.645458 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645435    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebc0553b-387f-438a-ab73-a053b37abcdd-serving-cert\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:14.645501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645446    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:14.645501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645480    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6lk59\""
Oct 30 20:37:14.645633 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645617    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-69hq7\""
Oct 30 20:37:14.645692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645672    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/395a48bf-cfce-443e-be13-b2939a9b9b8a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:14.645785 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645771    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.645868 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645841    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-49kbz\""
Oct 30 20:37:14.645921 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645897    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:37:14.645968 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.645918    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-trusted-ca\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:14.646109 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.646091    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/47697b9d-ed21-4e3b-93b0-320138d4037d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:14.646208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.646189    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebc0553b-387f-438a-ab73-a053b37abcdd-config\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:14.646305 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.646290    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/395a48bf-cfce-443e-be13-b2939a9b9b8a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:14.647283 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.647269    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.648075 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.648057    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.649703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.649691    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 20:37:14.649947 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.649927    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 20:37:14.649947 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.649932    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 20:37:14.650062 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.649948    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 20:37:14.650062 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.649981    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-q7pnc\""
Oct 30 20:37:14.650452 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.650441    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:37:14.650684 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.650671    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtwx7\""
Oct 30 20:37:14.651807 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.651792    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8wtn\" (UniqueName: \"kubernetes.io/projected/a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf-kube-api-access-c8wtn\") pod \"network-check-source-6f78d59f99-fhbw2\" (UID: \"a96bf55a-0e4c-4952-b7fc-4f20c3f84fdf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fhbw2"
Oct 30 20:37:14.653051 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.653036    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:37:14.654074 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.654060    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwsm\" (UniqueName: \"kubernetes.io/projected/7fc47c20-4a62-4167-9334-4d7b1cadb5b5-kube-api-access-dhwsm\") pod \"kas-connection-checker-685895c7c9-sp5fq\" (UID: \"7fc47c20-4a62-4167-9334-4d7b1cadb5b5\") " pod="kube-system/kas-connection-checker-685895c7c9-sp5fq"
Oct 30 20:37:14.654285 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.654267    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2m8\" (UniqueName: \"kubernetes.io/projected/8a707e7f-24ce-4322-b5ad-ffaaa21dc23c-kube-api-access-zr2m8\") pod \"kas-connection-checker-685895c7c9-jr5cw\" (UID: \"8a707e7f-24ce-4322-b5ad-ffaaa21dc23c\") " pod="kube-system/kas-connection-checker-685895c7c9-jr5cw"
Oct 30 20:37:14.654418 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.654400    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvvh\" (UniqueName: \"kubernetes.io/projected/ebc0553b-387f-438a-ab73-a053b37abcdd-kube-api-access-2jvvh\") pod \"console-operator-7cd56b8dbc-8nltv\" (UID: \"ebc0553b-387f-438a-ab73-a053b37abcdd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:14.654700 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.654676    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khjfm\" (UniqueName: \"kubernetes.io/projected/395a48bf-cfce-443e-be13-b2939a9b9b8a-kube-api-access-khjfm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n\" (UID: \"395a48bf-cfce-443e-be13-b2939a9b9b8a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n"
Oct 30 20:37:14.655044 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.655026    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjmlj\" (UniqueName: \"kubernetes.io/projected/f7aac07e-1e6f-4f62-897c-0061aa870749-kube-api-access-zjmlj\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:14.655164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.655150    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkh5r\" (UniqueName: \"kubernetes.io/projected/a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a-kube-api-access-pkh5r\") pod \"kas-connection-checker-685895c7c9-nbh76\" (UID: \"a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a\") " pod="kube-system/kas-connection-checker-685895c7c9-nbh76"
Oct 30 20:37:14.655279 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.655265    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtw88\" (UniqueName: \"kubernetes.io/projected/47697b9d-ed21-4e3b-93b0-320138d4037d-kube-api-access-dtw88\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:14.710536 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.710519    4810 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.725037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.725015    4810 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:37:14.725145 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.725059    4810 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.us-east-2.compute.internal"
Oct 30 20:37:14.725675 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.725660    4810 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:37:14.746082 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746066    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-stats-auth\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.746163 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746089    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bv5x\" (UniqueName: \"kubernetes.io/projected/7389168a-7f19-48df-98f8-6d6d20892416-kube-api-access-2bv5x\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:37:14.746163 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746112    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93f061fe-5c27-419b-8677-9734d5dce379-cert\") pod \"ingress-canary-x9v7g\" (UID: \"93f061fe-5c27-419b-8677-9734d5dce379\") " pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:14.746163 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746132    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-kubelet\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.746163 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746147    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.746360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746277    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-ovn\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.746360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746323    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysconfig\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.746360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746355    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-daemon-config\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.746505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746370    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-stats-auth\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.746505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746408    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2a1e731-e195-4a0b-b778-c606ccc13ef6-host-slash\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:37:14.746505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746487    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-etc-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.746617 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746502    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93f061fe-5c27-419b-8677-9734d5dce379-cert\") pod \"ingress-canary-x9v7g\" (UID: \"93f061fe-5c27-419b-8677-9734d5dce379\") " pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:14.746617 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746554    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6140cde0-b52b-48e1-a1d6-bcac35086922-config-volume\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:14.746617 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746582    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.746617 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746585    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.746748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746625    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51d19e04-9e59-4f4f-be5c-9900cca6d655-konnectivity-ca\") pod \"konnectivity-agent-vgcxk\" (UID: \"51d19e04-9e59-4f4f-be5c-9900cca6d655\") " pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:37:14.746748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746658    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-netns\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.746748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746699    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-daemon-config\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.746748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746703    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-etc-tuned\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.746748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746742    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-etc-kubernetes\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.746927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746766    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-etc-tuned\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.746927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746767    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.746927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746797    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e30414b5-8768-4eb1-a4ba-395115dba647-ovn-node-metrics-cert\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.746927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746822    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:14.746927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746848    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-trusted-ca\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.746927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746893    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-slash\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.746927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746920    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-trusted-ca-bundle\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:14.747119 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746942    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7b7j\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-kube-api-access-v7b7j\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.747119 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746965    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.747119 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746985    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6140cde0-b52b-48e1-a1d6-bcac35086922-config-volume\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:14.747119 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746994    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-system-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.747119 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.746798    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.747119 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747043    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51d19e04-9e59-4f4f-be5c-9900cca6d655-konnectivity-ca\") pod \"konnectivity-agent-vgcxk\" (UID: \"51d19e04-9e59-4f4f-be5c-9900cca6d655\") " pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:37:14.747273 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747249    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-trusted-ca-bundle\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:14.747302 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747266    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e30414b5-8768-4eb1-a4ba-395115dba647-ovn-node-metrics-cert\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.747363 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747337    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-trusted-ca\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.747417 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747392    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-key\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:14.747513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747428    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58b187ea-dead-4a37-8a52-5e447783d815-host\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:37:14.747513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747445    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2edfa6-676f-47ab-af10-37be9080c821-archives-path\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:14.747513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747462    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-systemd\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.747513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747480    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-sys\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.747513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747503    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.747747 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747520    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-bin\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.747747 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747544    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-socket-dir-parent\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.747747 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747560    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2edfa6-676f-47ab-af10-37be9080c821-archives-path\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:14.747747 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747566    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.747747 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747684    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:14.747986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747780    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-installation-pull-secrets\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.747986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747827    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-run\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.747986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747905    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cni-binary-copy\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.747986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747910    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-key\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:14.747986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747937    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-multus\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.747986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747961    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-tmp\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.747989    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-installation-pull-secrets\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748010    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-systemd-units\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748034    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-tmp\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748040    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-device-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748075    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-service-ca-bundle\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748084    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cni-binary-copy\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748109    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msb4h\" (UniqueName: \"kubernetes.io/projected/b3e2fc05-6adb-4c94-a124-aca4483a4e13-kube-api-access-msb4h\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748149    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8mz\" (UniqueName: \"kubernetes.io/projected/e30414b5-8768-4eb1-a4ba-395115dba647-kube-api-access-cj8mz\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748181    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3e2fc05-6adb-4c94-a124-aca4483a4e13-service-ca-bundle\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748179    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748193    4810 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748219    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4376a74-205b-45a1-b005-c59fae4f4201-config\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:14.748244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748246    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748271    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-default-certificate\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748295    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-netns\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748348    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-multus-certs\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748381    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-cabundle\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748409    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-env-overrides\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748450    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdc5\" (UniqueName: \"kubernetes.io/projected/72c43d73-596a-48d7-aa34-60c720b92fb6-kube-api-access-khdc5\") pod \"migrator-66df5d496b-vrlcq\" (UID: \"72c43d73-596a-48d7-aa34-60c720b92fb6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748474    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcxr\" (UniqueName: \"kubernetes.io/projected/2d2edfa6-676f-47ab-af10-37be9080c821-kube-api-access-xqcxr\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748508    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-default-certificate\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748507    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4376a74-205b-45a1-b005-c59fae4f4201-config\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748540    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-installation-pull-secrets\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748566    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-lib-modules\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748589    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-bin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748609    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-conf-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748632    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748634    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-env-overrides\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748656    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d7877a9-9cc5-4655-9b03-24fd447878ac-signing-cabundle\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748672    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfsd\" (UniqueName: \"kubernetes.io/projected/2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a-kube-api-access-wxfsd\") pod \"migrator-66df5d496b-xpdvf\" (UID: \"2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748713    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-installation-pull-secrets\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748716    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-image-registry-private-configuration\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748758    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jfd\" (UniqueName: \"kubernetes.io/projected/1d7877a9-9cc5-4655-9b03-24fd447878ac-kube-api-access-k2jfd\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:14.748825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748816    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-snapshots\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748841    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58b187ea-dead-4a37-8a52-5e447783d815-serviceca\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748885    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-conf\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748912    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748941    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4376a74-205b-45a1-b005-c59fae4f4201-serving-cert\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.748966    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-binary-copy\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749006    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-sys-fs\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749056    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b3e2fc05-6adb-4c94-a124-aca4483a4e13-snapshots\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749143    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-image-registry-private-configuration\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749216    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4376a74-205b-45a1-b005-c59fae4f4201-serving-cert\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749256    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlkc\" (UniqueName: \"kubernetes.io/projected/73fa532a-7546-4e60-8f8c-901cebfd7f07-kube-api-access-7zlkc\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749263    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-cni-binary-copy\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749276    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-var-lib-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749292    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749343    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthfq\" (UniqueName: \"kubernetes.io/projected/b4376a74-205b-45a1-b005-c59fae4f4201-kube-api-access-wthfq\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749360    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgww\" (UniqueName: \"kubernetes.io/projected/2ba0647d-e918-4bc7-843b-205e600536ec-kube-api-access-2qgww\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749378    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cnibin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749391    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-node-log\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749410    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749444    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plcbb\" (UniqueName: \"kubernetes.io/projected/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-kube-api-access-plcbb\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749524    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/734323fa-6946-426b-9e06-396cb4e7b48f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749579    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6140cde0-b52b-48e1-a1d6-bcac35086922-tmp-dir\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749576    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58b187ea-dead-4a37-8a52-5e447783d815-serviceca\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749677    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749702    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6140cde0-b52b-48e1-a1d6-bcac35086922-tmp-dir\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749717    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/734323fa-6946-426b-9e06-396cb4e7b48f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749748    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-var-lib-kubelet\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.749783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749772    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzzjh\" (UniqueName: \"kubernetes.io/projected/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-kube-api-access-fzzjh\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749838    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51d19e04-9e59-4f4f-be5c-9900cca6d655-agent-certs\") pod \"konnectivity-agent-vgcxk\" (UID: \"51d19e04-9e59-4f4f-be5c-9900cca6d655\") " pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749896    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749919    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-kubernetes\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749933    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-host\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.749950    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2a1e731-e195-4a0b-b778-c606ccc13ef6-iptables-alerter-script\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750086    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51d19e04-9e59-4f4f-be5c-9900cca6d655-agent-certs\") pod \"konnectivity-agent-vgcxk\" (UID: \"51d19e04-9e59-4f4f-be5c-9900cca6d655\") " pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750107    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94nz\" (UniqueName: \"kubernetes.io/projected/0d7898c1-9d6c-4382-baa2-e6baae34941a-kube-api-access-b94nz\") pod \"volume-data-source-validator-58c9f5f46d-klqns\" (UID: \"0d7898c1-9d6c-4382-baa2-e6baae34941a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750200    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-log-socket\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750199    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b2a1e731-e195-4a0b-b778-c606ccc13ef6-iptables-alerter-script\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750217    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-config\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750370    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-config\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750372    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-crio-socket\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750413    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-bound-sa-token\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750447    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-tmp\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750490    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-kubelet-config\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750511    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ba0647d-e918-4bc7-843b-205e600536ec-tmp\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750521    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-certificates\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750544    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtbb\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-kube-api-access-jvtbb\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.751020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750572    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-modprobe-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.750623    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751103    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-k8s-cni-cncf-io\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751134    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cwvl\" (UniqueName: \"kubernetes.io/projected/b2a1e731-e195-4a0b-b778-c606ccc13ef6-kube-api-access-7cwvl\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751155    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-socket-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751179    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751221    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7389168a-7f19-48df-98f8-6d6d20892416-tmp-dir\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751281    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-certificates\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751367    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrlf\" (UniqueName: \"kubernetes.io/projected/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-api-access-hqrlf\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751427    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7389168a-7f19-48df-98f8-6d6d20892416-tmp-dir\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751480    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxvc\" (UniqueName: \"kubernetes.io/projected/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-kube-api-access-qnxvc\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751502    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-image-registry-private-configuration\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751517    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dd510fb-f40e-401c-845f-23b9d750dc68-ca-trust-extracted\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751564    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/204ba748-a514-4772-9fba-dfafd3eddcf7-ca-trust-extracted\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751590    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751616    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmcl5\" (UniqueName: \"kubernetes.io/projected/6140cde0-b52b-48e1-a1d6-bcac35086922-kube-api-access-wmcl5\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751641    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-bound-sa-token\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751666    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751703    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-netd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751725    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-etc-selinux\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751725    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-image-registry-private-configuration\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751916    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8902c3bd-bf64-48e1-b113-d42ed8456348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751938    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/204ba748-a514-4772-9fba-dfafd3eddcf7-ca-trust-extracted\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751954    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.751985    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-dbus\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752001    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-kubelet\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752017    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-data-volume\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752034    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752153    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-registration-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752180    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a248c2d8-ba41-4190-9ca7-6cce702dd07a-original-pull-secret\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752207    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbbp\" (UniqueName: \"kubernetes.io/projected/58b187ea-dead-4a37-8a52-5e447783d815-kube-api-access-gpbbp\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752239    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-hostroot\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752266    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-system-cni-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752291    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-cnibin\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752343    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-systemd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752387    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752447    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xph2w\" (UniqueName: \"kubernetes.io/projected/8902c3bd-bf64-48e1-b113-d42ed8456348-kube-api-access-xph2w\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752482    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7aac07e-1e6f-4f62-897c-0061aa870749-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-crfsg\" (UID: \"f7aac07e-1e6f-4f62-897c-0061aa870749\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752490    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3e2fc05-6adb-4c94-a124-aca4483a4e13-serving-cert\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752524    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/47697b9d-ed21-4e3b-93b0-320138d4037d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cd75b\" (UID: \"47697b9d-ed21-4e3b-93b0-320138d4037d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752525    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-service-ca-bundle\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752563    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-trusted-ca\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752579    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfnr\" (UniqueName: \"kubernetes.io/projected/93f061fe-5c27-419b-8677-9734d5dce379-kube-api-access-scfnr\") pod \"ingress-canary-x9v7g\" (UID: \"93f061fe-5c27-419b-8677-9734d5dce379\") " pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752598    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-certificates\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752651    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-service-ca-bundle\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752752    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-script-lib\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752782    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7389168a-7f19-48df-98f8-6d6d20892416-hosts-file\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752904    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3e2fc05-6adb-4c94-a124-aca4483a4e13-serving-cert\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752971    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-trusted-ca\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752979    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e30414b5-8768-4eb1-a4ba-395115dba647-ovnkube-script-lib\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.752979    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-data-volume\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.753169    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-certificates\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.754207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.753959    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dd510fb-f40e-401c-845f-23b9d750dc68-ca-trust-extracted\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.756481 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.754672    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bv5x\" (UniqueName: \"kubernetes.io/projected/7389168a-7f19-48df-98f8-6d6d20892416-kube-api-access-2bv5x\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:37:14.756481 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.755468    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7b7j\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-kube-api-access-v7b7j\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.756481 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.756007    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcxr\" (UniqueName: \"kubernetes.io/projected/2d2edfa6-676f-47ab-af10-37be9080c821-kube-api-access-xqcxr\") pod \"periodic-gathering-7k6rz-jhhmx\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") " pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:37:14.756481 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.756062    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msb4h\" (UniqueName: \"kubernetes.io/projected/b3e2fc05-6adb-4c94-a124-aca4483a4e13-kube-api-access-msb4h\") pod \"insights-operator-7c4986d449-rbf5z\" (UID: \"b3e2fc05-6adb-4c94-a124-aca4483a4e13\") " pod="openshift-insights/insights-operator-7c4986d449-rbf5z"
Oct 30 20:37:14.763249 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.763230    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfsd\" (UniqueName: \"kubernetes.io/projected/2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a-kube-api-access-wxfsd\") pod \"migrator-66df5d496b-xpdvf\" (UID: \"2cc5b9cd-7d9f-41e9-aa64-aafdc750f62a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xpdvf"
Oct 30 20:37:14.763413 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.763397    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jfd\" (UniqueName: \"kubernetes.io/projected/1d7877a9-9cc5-4655-9b03-24fd447878ac-kube-api-access-k2jfd\") pod \"service-ca-5fffc97bdf-zqt6s\" (UID: \"1d7877a9-9cc5-4655-9b03-24fd447878ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zqt6s"
Oct 30 20:37:14.766347 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.766330    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qgww\" (UniqueName: \"kubernetes.io/projected/2ba0647d-e918-4bc7-843b-205e600536ec-kube-api-access-2qgww\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.771823 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.771800    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zlkc\" (UniqueName: \"kubernetes.io/projected/73fa532a-7546-4e60-8f8c-901cebfd7f07-kube-api-access-7zlkc\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.772002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.771986    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8mz\" (UniqueName: \"kubernetes.io/projected/e30414b5-8768-4eb1-a4ba-395115dba647-kube-api-access-cj8mz\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.772165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.772148    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdc5\" (UniqueName: \"kubernetes.io/projected/72c43d73-596a-48d7-aa34-60c720b92fb6-kube-api-access-khdc5\") pod \"migrator-66df5d496b-vrlcq\" (UID: \"72c43d73-596a-48d7-aa34-60c720b92fb6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vrlcq"
Oct 30 20:37:14.773849 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.773810    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ksf\" (UniqueName: \"kubernetes.io/projected/2fc2fd6c-3d77-4446-9402-cf73cdd3497a-kube-api-access-m8ksf\") pod \"network-check-target-lfl79\" (UID: \"2fc2fd6c-3d77-4446-9402-cf73cdd3497a\") " pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:37:14.774093 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.774075    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthfq\" (UniqueName: \"kubernetes.io/projected/b4376a74-205b-45a1-b005-c59fae4f4201-kube-api-access-wthfq\") pod \"service-ca-operator-544d749c8c-75fxd\" (UID: \"b4376a74-205b-45a1-b005-c59fae4f4201\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd"
Oct 30 20:37:14.774928 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.774910    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plcbb\" (UniqueName: \"kubernetes.io/projected/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-kube-api-access-plcbb\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.775014 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.774978    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxvc\" (UniqueName: \"kubernetes.io/projected/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-kube-api-access-qnxvc\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:37:14.779176 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.775491    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmcl5\" (UniqueName: \"kubernetes.io/projected/6140cde0-b52b-48e1-a1d6-bcac35086922-kube-api-access-wmcl5\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:14.779176 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.775646    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbbp\" (UniqueName: \"kubernetes.io/projected/58b187ea-dead-4a37-8a52-5e447783d815-kube-api-access-gpbbp\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:37:14.779176 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.775727    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvtbb\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-kube-api-access-jvtbb\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.780087 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.779823    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-bound-sa-token\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.780087 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.779895    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cwvl\" (UniqueName: \"kubernetes.io/projected/b2a1e731-e195-4a0b-b778-c606ccc13ef6-kube-api-access-7cwvl\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:37:14.780087 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.780037    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzzjh\" (UniqueName: \"kubernetes.io/projected/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-kube-api-access-fzzjh\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.780293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.780188    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94nz\" (UniqueName: \"kubernetes.io/projected/0d7898c1-9d6c-4382-baa2-e6baae34941a-kube-api-access-b94nz\") pod \"volume-data-source-validator-58c9f5f46d-klqns\" (UID: \"0d7898c1-9d6c-4382-baa2-e6baae34941a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-klqns"
Oct 30 20:37:14.780343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.780286    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-bound-sa-token\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.780343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.780293    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrlf\" (UniqueName: \"kubernetes.io/projected/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-api-access-hqrlf\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.780436 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.780377    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfnr\" (UniqueName: \"kubernetes.io/projected/93f061fe-5c27-419b-8677-9734d5dce379-kube-api-access-scfnr\") pod \"ingress-canary-x9v7g\" (UID: \"93f061fe-5c27-419b-8677-9734d5dce379\") " pod="openshift-ingress-canary/ingress-canary-x9v7g"
Oct 30 20:37:14.781505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.781480    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xph2w\" (UniqueName: \"kubernetes.io/projected/8902c3bd-bf64-48e1-b113-d42ed8456348-kube-api-access-xph2w\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.792152 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.792116    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:14.853216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853194    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-kubernetes\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.853216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853218    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-host\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.853376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853233    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-log-socket\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.853376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853250    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-crio-socket\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.853376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853267    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-kubelet-config\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:14.853376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853317    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-host\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.853376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853326    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-kubelet-config\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:14.853376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853325    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-log-socket\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.853376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853337    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-kubernetes\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.853376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853364    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-modprobe-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.853376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853369    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-crio-socket\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853394    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853421    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-k8s-cni-cncf-io\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853446    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-socket-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853460    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-modprobe-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853463    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853469    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853498    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-k8s-cni-cncf-io\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853510    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-netd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853532    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-etc-selinux\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853547    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853532    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-netd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.853559    4810 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853568    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-etc-selinux\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853593    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-socket-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.853626    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:15.35360619 +0000 UTC m=+1.845422731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : secret "router-metrics-certs-default" not found
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853653    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-dbus\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853675    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-kubelet\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853697    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853716    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a248c2d8-ba41-4190-9ca7-6cce702dd07a-dbus\") pod \"global-pull-secret-syncer-vmcq2\" (UID: \"a248c2d8-ba41-4190-9ca7-6cce702dd07a\") " pod="kube-system/global-pull-secret-syncer-vmcq2"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853719    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-registration-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853742    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-kubelet\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853745    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-hostroot\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853779    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-hostroot\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853780    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-system-cni-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853810    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-system-cni-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853813    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-registration-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.853803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853816    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-cnibin\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853843    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-cnibin\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853877    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-systemd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.853819    4810 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853901    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.853928    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls podName:6140cde0-b52b-48e1-a1d6-bcac35086922 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:15.353916746 +0000 UTC m=+1.845733269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls") pod "dns-default-nfvzn" (UID: "6140cde0-b52b-48e1-a1d6-bcac35086922") : secret "dns-default-metrics-tls" not found
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853947    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7389168a-7f19-48df-98f8-6d6d20892416-hosts-file\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853947    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-systemd\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853963    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-kubelet\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853987    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-ovn\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.853992    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-d\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854010    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysconfig\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854009    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-kubelet\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854025    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2a1e731-e195-4a0b-b778-c606ccc13ef6-host-slash\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854032    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-ovn\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854039    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-etc-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854008    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7389168a-7f19-48df-98f8-6d6d20892416-hosts-file\") pod \"node-resolver-4mv46\" (UID: \"7389168a-7f19-48df-98f8-6d6d20892416\") " pod="openshift-dns/node-resolver-4mv46"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854063    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-netns\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854064    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysconfig\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854078    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-etc-kubernetes\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854095    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854110    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-netns\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854111    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854077    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b2a1e731-e195-4a0b-b778-c606ccc13ef6-host-slash\") pod \"iptables-alerter-nm42c\" (UID: \"b2a1e731-e195-4a0b-b778-c606ccc13ef6\") " pod="openshift-network-operator/iptables-alerter-nm42c"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854121    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854110    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-etc-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854158    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-etc-kubernetes\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854232    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8902c3bd-bf64-48e1-b113-d42ed8456348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-88xhl\" (UID: \"8902c3bd-bf64-48e1-b113-d42ed8456348\") " pod="openshift-multus/multus-additional-cni-plugins-88xhl"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854252    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-slash\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854274    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854291    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-system-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854306    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58b187ea-dead-4a37-8a52-5e447783d815-host\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854320    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-systemd\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854341    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-sys\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854364    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-slash\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854377    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58b187ea-dead-4a37-8a52-5e447783d815-host\") pod \"node-ca-sq5lx\" (UID: \"58b187ea-dead-4a37-8a52-5e447783d815\") " pod="openshift-image-registry/node-ca-sq5lx"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854365    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854409    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-systemd\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854414    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854420    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-bin\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854411    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-system-cni-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854426    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-sys\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.854386    4810 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854446    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-socket-dir-parent\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.854457    4810 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-948b95b59-tdzdl: secret "image-registry-tls" not found
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854470    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854450    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-cni-bin\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854492    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-run\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.854519    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls podName:4dd510fb-f40e-401c-845f-23b9d750dc68 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:15.354504468 +0000 UTC m=+1.846321012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls") pod "image-registry-948b95b59-tdzdl" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68") : secret "image-registry-tls" not found
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854520    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-socket-dir-parent\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854535    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-multus\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854542    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-run\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854559    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-systemd-units\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854579    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-device-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854588    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-multus\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.854592    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:15.354576597 +0000 UTC m=+1.846393136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : configmap references non-existent config key: service-ca.crt
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854614    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-device-dir\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854634    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854641    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-systemd-units\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854659    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854692    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-netns\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854714    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-multus-certs\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.854740    4810 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854792    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-netns\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854740    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-lib-modules\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.854821    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:15.354804583 +0000 UTC m=+1.846621140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854830    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-lib-modules\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854835    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-run-multus-certs\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854847    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-bin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854891    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-conf-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854911    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-conf\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854933    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-host-var-lib-cni-bin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854968    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-multus-conf-dir\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854969    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.854991    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855008    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-sys-fs\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855033    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-var-lib-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.854985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855039    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-etc-sysctl-conf\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855055    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855081    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cnibin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855092    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73fa532a-7546-4e60-8f8c-901cebfd7f07-sys-fs\") pod \"aws-ebs-csi-driver-node-z54xs\" (UID: \"73fa532a-7546-4e60-8f8c-901cebfd7f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z54xs"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855110    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-run-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855117    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-var-lib-openvswitch\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855102    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-node-log\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855136    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8d37e01-bf52-40e0-bc5a-524ce20ddad4-cnibin\") pod \"multus-clqzg\" (UID: \"d8d37e01-bf52-40e0-bc5a-524ce20ddad4\") " pod="openshift-multus/multus-clqzg"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855154    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855172    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-node-log\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855184    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855220    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-var-lib-kubelet\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855242    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30414b5-8768-4eb1-a4ba-395115dba647-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4spg\" (UID: \"e30414b5-8768-4eb1-a4ba-395115dba647\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.855254    4810 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.855267    4810 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfdcc7f98-zlptz: secret "image-registry-tls" not found
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.855279    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ba0647d-e918-4bc7-843b-205e600536ec-var-lib-kubelet\") pod \"tuned-bkx7g\" (UID: \"2ba0647d-e918-4bc7-843b-205e600536ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bkx7g"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:14.855318    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls podName:204ba748-a514-4772-9fba-dfafd3eddcf7 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:15.355304299 +0000 UTC m=+1.847120836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls") pod "image-registry-5bfdcc7f98-zlptz" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7") : secret "image-registry-tls" not found
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.856948    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efcea4c-b6b6-47a5-be1d-c61a83bbe907-metrics-certs\") pod \"network-metrics-daemon-gtkw8\" (UID: \"3efcea4c-b6b6-47a5-be1d-c61a83bbe907\") " pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:37:14.857829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.856948    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/734323fa-6946-426b-9e06-396cb4e7b48f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wt7nr\" (UID: \"734323fa-6946-426b-9e06-396cb4e7b48f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:14.903143 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.903123    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:14.923187 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.923168    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"
Oct 30 20:37:14.924001 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.923983    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"
Oct 30 20:37:14.924071 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.924052    4810 scope.go:122] "RemoveContainer" containerID="f354ab5f6dc4cbe1be1a150d8a98d993f7b1d3c8cf4857d96e6ad15eecfa725f"
Oct 30 20:37:14.932025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.932003    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"
Oct 30 20:37:14.933599 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:14.933582    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gtkw8"
Oct 30 20:37:15.225388 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.225168    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg"]
Oct 30 20:37:15.266172 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.266151    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b"]
Oct 30 20:37:15.270149 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:15.270119    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47697b9d_ed21_4e3b_93b0_320138d4037d.slice/crio-6d33ffaed61521c14f6ba38ceab7facb775436440bfeb2592febe8d0950721fc WatchSource:0}: Error finding container 6d33ffaed61521c14f6ba38ceab7facb775436440bfeb2592febe8d0950721fc: Status 404 returned error can't find the container with id 6d33ffaed61521c14f6ba38ceab7facb775436440bfeb2592febe8d0950721fc
Oct 30 20:37:15.272195 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.272178    4810 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:37:15.286018 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.285996    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gtkw8"]
Oct 30 20:37:15.289545 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:15.289519    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efcea4c_b6b6_47a5_be1d_c61a83bbe907.slice/crio-ec9040c118aeaacd07f2c21a371fbb5583b4f3ca927dd32a7c3983515f79b27b WatchSource:0}: Error finding container ec9040c118aeaacd07f2c21a371fbb5583b4f3ca927dd32a7c3983515f79b27b: Status 404 returned error can't find the container with id ec9040c118aeaacd07f2c21a371fbb5583b4f3ca927dd32a7c3983515f79b27b
Oct 30 20:37:15.309614 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.309592    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wt7nr"]
Oct 30 20:37:15.312766 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:15.312742    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734323fa_6946_426b_9e06_396cb4e7b48f.slice/crio-7586fd79652c82a64ba8e3a3dae7beb4ab1c7eb153fada2db556bbedeae467d8 WatchSource:0}: Error finding container 7586fd79652c82a64ba8e3a3dae7beb4ab1c7eb153fada2db556bbedeae467d8: Status 404 returned error can't find the container with id 7586fd79652c82a64ba8e3a3dae7beb4ab1c7eb153fada2db556bbedeae467d8
Oct 30 20:37:15.360679 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.360602    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:15.360679 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.360652    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.360689    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.360741    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:15.360778    4810 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:15.360838    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:16.360823317 +0000 UTC m=+2.852639840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : secret "router-metrics-certs-default" not found
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:15.360842    4810 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.360777    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:15.360886    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle podName:6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:16.360844472 +0000 UTC m=+2.852660995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle") pod "router-default-7dc99f8759-pzknq" (UID: "6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2") : configmap references non-existent config key: service-ca.crt
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:15.360912    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls podName:6140cde0-b52b-48e1-a1d6-bcac35086922 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:16.360901951 +0000 UTC m=+2.852718477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls") pod "dns-default-nfvzn" (UID: "6140cde0-b52b-48e1-a1d6-bcac35086922") : secret "dns-default-metrics-tls" not found
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.360942    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:15.361030    4810 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:15.361106 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:15.361075    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:16.361060921 +0000 UTC m=+2.852877465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:15.364073 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.364054    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"image-registry-5bfdcc7f98-zlptz\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") " pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:15.364154 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.364057    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"image-registry-948b95b59-tdzdl\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") " pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:15.427185 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.427139    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:15.525889 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.525818    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:15.538158 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.538135    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:15.718171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.718147    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:37:15.718989 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.718614    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/1.log"
Oct 30 20:37:15.718989 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.718642    4810 generic.go:356] "Generic (PLEG): container finished" podID="ebc0553b-387f-438a-ab73-a053b37abcdd" containerID="c19a41d4c92faed2db6131f82696629dd7f32d3cb75fbe4668c6a38ddec4a106" exitCode=255
Oct 30 20:37:15.718989 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.718712    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" event={"ID":"ebc0553b-387f-438a-ab73-a053b37abcdd","Type":"ContainerDied","Data":"c19a41d4c92faed2db6131f82696629dd7f32d3cb75fbe4668c6a38ddec4a106"}
Oct 30 20:37:15.718989 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.718786    4810 scope.go:122] "RemoveContainer" containerID="f354ab5f6dc4cbe1be1a150d8a98d993f7b1d3c8cf4857d96e6ad15eecfa725f"
Oct 30 20:37:15.719210 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.719109    4810 scope.go:122] "RemoveContainer" containerID="c19a41d4c92faed2db6131f82696629dd7f32d3cb75fbe4668c6a38ddec4a106"
Oct 30 20:37:15.719370 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:15.719346    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8nltv_openshift-console-operator(ebc0553b-387f-438a-ab73-a053b37abcdd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" podUID="ebc0553b-387f-438a-ab73-a053b37abcdd"
Oct 30 20:37:15.719966 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.719946    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gtkw8" event={"ID":"3efcea4c-b6b6-47a5-be1d-c61a83bbe907","Type":"ContainerStarted","Data":"ec9040c118aeaacd07f2c21a371fbb5583b4f3ca927dd32a7c3983515f79b27b"}
Oct 30 20:37:15.721125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.721106    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b" event={"ID":"47697b9d-ed21-4e3b-93b0-320138d4037d","Type":"ContainerStarted","Data":"6d33ffaed61521c14f6ba38ceab7facb775436440bfeb2592febe8d0950721fc"}
Oct 30 20:37:15.722126 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.722110    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg" event={"ID":"f7aac07e-1e6f-4f62-897c-0061aa870749","Type":"ContainerStarted","Data":"41f4b414a9dfc2e09203221fc9bbcba7bddc9545a890acdafc2a2fd6edada747"}
Oct 30 20:37:15.723336 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.723315    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr" event={"ID":"734323fa-6946-426b-9e06-396cb4e7b48f","Type":"ContainerStarted","Data":"7586fd79652c82a64ba8e3a3dae7beb4ab1c7eb153fada2db556bbedeae467d8"}
Oct 30 20:37:15.758954 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.758929    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:15.812384 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.812198    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bfdcc7f98-zlptz"]
Oct 30 20:37:15.815574 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:15.815553    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204ba748_a514_4772_9fba_dfafd3eddcf7.slice/crio-f4e223ec55c14e792883a477e3416314f4b70dff96cb11cf7d0d84c0402b0bd9 WatchSource:0}: Error finding container f4e223ec55c14e792883a477e3416314f4b70dff96cb11cf7d0d84c0402b0bd9: Status 404 returned error can't find the container with id f4e223ec55c14e792883a477e3416314f4b70dff96cb11cf7d0d84c0402b0bd9
Oct 30 20:37:15.853844 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.853617    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-948b95b59-tdzdl"]
Oct 30 20:37:15.863815 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:15.863788    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd510fb_f40e_401c_845f_23b9d750dc68.slice/crio-c75a50aeb54ce34fc6bf3a3689711c09825c1ff790391b0e2d031c0e94530423 WatchSource:0}: Error finding container c75a50aeb54ce34fc6bf3a3689711c09825c1ff790391b0e2d031c0e94530423: Status 404 returned error can't find the container with id c75a50aeb54ce34fc6bf3a3689711c09825c1ff790391b0e2d031c0e94530423
Oct 30 20:37:15.953281 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:15.953247    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.150430 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.150397    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.355491 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.355454    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.385365 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.385335    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:16.393612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.385384    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:16.393612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.385442    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:16.393612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.385482    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:16.393612 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:16.385655    4810 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:37:16.393612 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:16.385713    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls podName:6140cde0-b52b-48e1-a1d6-bcac35086922 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:18.385696923 +0000 UTC m=+4.877513465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls") pod "dns-default-nfvzn" (UID: "6140cde0-b52b-48e1-a1d6-bcac35086922") : secret "dns-default-metrics-tls" not found
Oct 30 20:37:16.393612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.386439    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-service-ca-bundle\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:16.393612 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:16.386538    4810 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:16.393612 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:16.386584    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:18.386569228 +0000 UTC m=+4.878385760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:16.393612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.390417    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2-metrics-certs\") pod \"router-default-7dc99f8759-pzknq\" (UID: \"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2\") " pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:16.394264 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.394192    4810 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:16.430158 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.429968    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:16.459310 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.459194    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.558432 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.557952    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.657995 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.657948    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.729442 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.729250    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-948b95b59-tdzdl" event={"ID":"4dd510fb-f40e-401c-845f-23b9d750dc68","Type":"ContainerStarted","Data":"668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0"}
Oct 30 20:37:16.729442 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.729307    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-948b95b59-tdzdl" event={"ID":"4dd510fb-f40e-401c-845f-23b9d750dc68","Type":"ContainerStarted","Data":"c75a50aeb54ce34fc6bf3a3689711c09825c1ff790391b0e2d031c0e94530423"}
Oct 30 20:37:16.733014 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.732699    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:37:16.733152 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.733136    4810 scope.go:122] "RemoveContainer" containerID="c19a41d4c92faed2db6131f82696629dd7f32d3cb75fbe4668c6a38ddec4a106"
Oct 30 20:37:16.733324 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:16.733305    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8nltv_openshift-console-operator(ebc0553b-387f-438a-ab73-a053b37abcdd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" podUID="ebc0553b-387f-438a-ab73-a053b37abcdd"
Oct 30 20:37:16.737885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.737617    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz" event={"ID":"204ba748-a514-4772-9fba-dfafd3eddcf7","Type":"ContainerStarted","Data":"fdbeaea20112a0a8345b6226da3a79e3987b2360a641c1eb15ef28c628c7a6b0"}
Oct 30 20:37:16.737885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.737659    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz" event={"ID":"204ba748-a514-4772-9fba-dfafd3eddcf7","Type":"ContainerStarted","Data":"f4e223ec55c14e792883a477e3416314f4b70dff96cb11cf7d0d84c0402b0bd9"}
Oct 30 20:37:16.756559 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.756391    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.761968 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.761902    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7dc99f8759-pzknq"]
Oct 30 20:37:16.771793 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:16.771766    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0e6adb_30a6_481c_8ee0_27c52c7c2fe2.slice/crio-e3d08c248360cd8fe3ce253c24b0e573b313c96be2dbf2b0526a045d5341c2cf WatchSource:0}: Error finding container e3d08c248360cd8fe3ce253c24b0e573b313c96be2dbf2b0526a045d5341c2cf: Status 404 returned error can't find the container with id e3d08c248360cd8fe3ce253c24b0e573b313c96be2dbf2b0526a045d5341c2cf
Oct 30 20:37:16.839342 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:16.839311    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:17.743974 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:17.743721    4810 generic.go:356] "Generic (PLEG): container finished" podID="395a48bf-cfce-443e-be13-b2939a9b9b8a" containerID="e25786359b81e42d236510c30175f63a6035115b38804dc067d64f993f71ee71" exitCode=0
Oct 30 20:37:17.744432 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:17.743805    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n" event={"ID":"395a48bf-cfce-443e-be13-b2939a9b9b8a","Type":"ContainerDied","Data":"e25786359b81e42d236510c30175f63a6035115b38804dc067d64f993f71ee71"}
Oct 30 20:37:17.744432 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:17.744420    4810 scope.go:122] "RemoveContainer" containerID="e25786359b81e42d236510c30175f63a6035115b38804dc067d64f993f71ee71"
Oct 30 20:37:17.745969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:17.745885    4810 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:37:17.745969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:17.745923    4810 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:37:17.746143 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:17.745885    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7dc99f8759-pzknq" event={"ID":"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2","Type":"ContainerStarted","Data":"c4389eb5ef87730ec18a729444127a570fbbc38628d696454704f42864b2d5c2"}
Oct 30 20:37:17.746143 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:17.746029    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7dc99f8759-pzknq" event={"ID":"6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2","Type":"ContainerStarted","Data":"e3d08c248360cd8fe3ce253c24b0e573b313c96be2dbf2b0526a045d5341c2cf"}
Oct 30 20:37:18.048874 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.048766    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:18.198190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.198165    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:18.412575 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.407673    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:18.412575 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.407988    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:18.412575 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:18.408137    4810 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:18.412575 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:18.408183    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls podName:585e93ee-0fc2-4461-b5ad-e9e0f50afc06 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:22.408170664 +0000 UTC m=+8.899987200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kz7s8" (UID: "585e93ee-0fc2-4461-b5ad-e9e0f50afc06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:37:18.413726 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.413461    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6140cde0-b52b-48e1-a1d6-bcac35086922-metrics-tls\") pod \"dns-default-nfvzn\" (UID: \"6140cde0-b52b-48e1-a1d6-bcac35086922\") " pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:18.431337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.431245    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:18.436980 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.436954    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:18.447664 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.447634    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:18.541216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.541183    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:18.553846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.553824    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:18.656276 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.656242    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:18.752690 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.752655    4810 generic.go:356] "Generic (PLEG): container finished" podID="b3e2fc05-6adb-4c94-a124-aca4483a4e13" containerID="5920e6d7de20c10a6942708c81db189a50991832bcff606858220d8dcd1b7e33" exitCode=0
Oct 30 20:37:18.753572 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.752731    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rbf5z" event={"ID":"b3e2fc05-6adb-4c94-a124-aca4483a4e13","Type":"ContainerDied","Data":"5920e6d7de20c10a6942708c81db189a50991832bcff606858220d8dcd1b7e33"}
Oct 30 20:37:18.753572 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.753153    4810 scope.go:122] "RemoveContainer" containerID="5920e6d7de20c10a6942708c81db189a50991832bcff606858220d8dcd1b7e33"
Oct 30 20:37:18.755248 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.755225    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-z5f5n" event={"ID":"395a48bf-cfce-443e-be13-b2939a9b9b8a","Type":"ContainerStarted","Data":"3789997a2df78caba7f1860e490cd20fa7c76db51a845525dd85c63ff41ed5f8"}
Oct 30 20:37:18.759578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.757346    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:18.759578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.758819    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gtkw8" event={"ID":"3efcea4c-b6b6-47a5-be1d-c61a83bbe907","Type":"ContainerStarted","Data":"b8fa56aa23b1c774831eaf73cd72659f696c5231716e929d0815ecdb0d4f1ca1"}
Oct 30 20:37:18.759578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.758873    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gtkw8" event={"ID":"3efcea4c-b6b6-47a5-be1d-c61a83bbe907","Type":"ContainerStarted","Data":"cf777eab964f52a59e5e2317bcab6fdd29ac2f29c1c098c4493dc204eee188b6"}
Oct 30 20:37:18.761192 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.760876    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cd75b" event={"ID":"47697b9d-ed21-4e3b-93b0-320138d4037d","Type":"ContainerStarted","Data":"aa301475e5602cc0f79b7a9c7c509a409c830802b146409781719f9da2bc0dba"}
Oct 30 20:37:18.767376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.767350    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg" event={"ID":"f7aac07e-1e6f-4f62-897c-0061aa870749","Type":"ContainerStarted","Data":"eda2b10b6d3d626b9ef5bb5bb6e659ba1a958f85ecb4ce52966ac5244c2dcbcc"}
Oct 30 20:37:18.767479 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.767399    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-crfsg" event={"ID":"f7aac07e-1e6f-4f62-897c-0061aa870749","Type":"ContainerStarted","Data":"1fe6004d81e61faa19eac8cfa94f702a9b03130223c8bf6c640949ce544c17fc"}
Oct 30 20:37:18.770089 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.770055    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wt7nr" event={"ID":"734323fa-6946-426b-9e06-396cb4e7b48f","Type":"ContainerStarted","Data":"66ef7f7360cd7bbf243c704aef063aba1d29be2d44b078fed4de17e0089db3b4"}
Oct 30 20:37:18.831035 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.831008    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nfvzn"]
Oct 30 20:37:18.833843 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:18.833817    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6140cde0_b52b_48e1_a1d6_bcac35086922.slice/crio-9b7313661ef55f1fb92b432b2020d00c5b26f4042ee619323e1adb01f5a90b8d WatchSource:0}: Error finding container 9b7313661ef55f1fb92b432b2020d00c5b26f4042ee619323e1adb01f5a90b8d: Status 404 returned error can't find the container with id 9b7313661ef55f1fb92b432b2020d00c5b26f4042ee619323e1adb01f5a90b8d
Oct 30 20:37:18.858658 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.858623    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:18.886965 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:18.886925    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:19.050516 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:19.050410    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:19.776712 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:19.776668    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rbf5z" event={"ID":"b3e2fc05-6adb-4c94-a124-aca4483a4e13","Type":"ContainerStarted","Data":"79ac9a599cf3242c80b15d85d7c35581883bed2757c24df1d5a15483a7b2e57b"}
Oct 30 20:37:19.778203 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:19.778175    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfvzn" event={"ID":"6140cde0-b52b-48e1-a1d6-bcac35086922","Type":"ContainerStarted","Data":"9b7313661ef55f1fb92b432b2020d00c5b26f4042ee619323e1adb01f5a90b8d"}
Oct 30 20:37:19.778537 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:19.778516    4810 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:37:19.943551 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:19.943514    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:20.144708 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:20.144675    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:20.349792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:20.349758    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:20.488706 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:20.488676    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:20.488930 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:20.488914    4810 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:37:20.568064 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:20.568030    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:20.757062 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:20.756980    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:20.784340 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:20.784305    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfvzn" event={"ID":"6140cde0-b52b-48e1-a1d6-bcac35086922","Type":"ContainerStarted","Data":"460221dcd189d69649ca7d9c75c2562bd98b1a72c1f0e2fb67c7d1d3f9ac24e1"}
Oct 30 20:37:20.784723 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:20.784393    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfvzn" event={"ID":"6140cde0-b52b-48e1-a1d6-bcac35086922","Type":"ContainerStarted","Data":"df812f0d9f5e40339308def035fc6d4f074609491adab201907ed29d36020aeb"}
Oct 30 20:37:20.784723 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:20.784424    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:20.847893 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:20.847847    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:21.156334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:21.156307    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:22.448361 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:22.448321    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:22.450591 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:22.450565    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/585e93ee-0fc2-4461-b5ad-e9e0f50afc06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kz7s8\" (UID: \"585e93ee-0fc2-4461-b5ad-e9e0f50afc06\") " pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:22.747753 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:22.747673    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kz7s8"
Oct 30 20:37:23.017936 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:23.017850    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kz7s8"]
Oct 30 20:37:23.021518 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:23.021489    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585e93ee_0fc2_4461_b5ad_e9e0f50afc06.slice/crio-60e0b75b2ccbfb0d80e0eeca94189c52c68cdfae6745a7a2277006f9f1828a9a WatchSource:0}: Error finding container 60e0b75b2ccbfb0d80e0eeca94189c52c68cdfae6745a7a2277006f9f1828a9a: Status 404 returned error can't find the container with id 60e0b75b2ccbfb0d80e0eeca94189c52c68cdfae6745a7a2277006f9f1828a9a
Oct 30 20:37:23.540251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:23.540217    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:23.797808 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:23.797785    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kz7s8" event={"ID":"585e93ee-0fc2-4461-b5ad-e9e0f50afc06","Type":"ContainerStarted","Data":"04740c7876d00250b82f6f3ec583c9b080ba97aba68b34782919c25f2ea6ff4c"}
Oct 30 20:37:23.797925 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:23.797842    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kz7s8" event={"ID":"585e93ee-0fc2-4461-b5ad-e9e0f50afc06","Type":"ContainerStarted","Data":"60e0b75b2ccbfb0d80e0eeca94189c52c68cdfae6745a7a2277006f9f1828a9a"}
Oct 30 20:37:23.802958 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:23.802935    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:23.803293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:23.803280    4810 scope.go:122] "RemoveContainer" containerID="c19a41d4c92faed2db6131f82696629dd7f32d3cb75fbe4668c6a38ddec4a106"
Oct 30 20:37:23.803470 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:23.803450    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8nltv_openshift-console-operator(ebc0553b-387f-438a-ab73-a053b37abcdd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" podUID="ebc0553b-387f-438a-ab73-a053b37abcdd"
Oct 30 20:37:23.912286 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:23.912260    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:23.912432 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:23.912387    4810 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:37:23.913547 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:23.913522    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7dc99f8759-pzknq"
Oct 30 20:37:24.718706 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:24.718674    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:37:24.803566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:24.803524    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kz7s8" event={"ID":"585e93ee-0fc2-4461-b5ad-e9e0f50afc06","Type":"ContainerStarted","Data":"458163186d64f3caaa41306a4c0a816e365a84027618eecce0bba4db71b0148c"}
Oct 30 20:37:25.159041 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:25.159013    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:25.654838 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:25.654811    4810 scope.go:122] "RemoveContainer" containerID="c19a41d4c92faed2db6131f82696629dd7f32d3cb75fbe4668c6a38ddec4a106"
Oct 30 20:37:25.809267 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:25.809246    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:37:25.809675 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:25.809332    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv" event={"ID":"ebc0553b-387f-438a-ab73-a053b37abcdd","Type":"ContainerStarted","Data":"7a32ac2e3a49f4e9b82faa8534951bc4f47771fa1d2fa7a54a8514973f4bb3f7"}
Oct 30 20:37:25.810033 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:25.809733    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:25.811720 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:25.811696    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kz7s8" event={"ID":"585e93ee-0fc2-4461-b5ad-e9e0f50afc06","Type":"ContainerStarted","Data":"e2230a2f815e0e2747547d8a0850d9f8b257a8e9b46d9d88c53c06cf7aa500bb"}
Oct 30 20:37:25.959427 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:25.959364    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8nltv"
Oct 30 20:37:26.212660 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:26.212583    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:26.323318 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:26.323234    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:26.432300 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:26.432267    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:26.663596 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:26.663567    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:26.785675 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:26.785645    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:27.271792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:27.271766    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vgcxk"
Oct 30 20:37:27.673786 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:27.673755    4810 patch_prober.go:29] interesting pod/image-registry-5bfdcc7f98-zlptz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:37:27.673946 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:27.673801    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz" podUID="204ba748-a514-4772-9fba-dfafd3eddcf7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:37:28.504755 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:28.504713    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:28.580657 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:28.580610    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:29.880294 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.880262    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-g57td"]
Oct 30 20:37:29.883828 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.883806    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-g57td"
Oct 30 20:37:29.887200 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.887178    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-x48bh-vgjns"]
Oct 30 20:37:29.890146 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.890101    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:29.895024 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.895003    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 20:37:29.895105 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.895079    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 20:37:29.895226 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.895206    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-hmbb2\""
Oct 30 20:37:29.895572 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.895552    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bfdcc7f98-zlptz"]
Oct 30 20:37:29.900145 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.900105    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-g57td"]
Oct 30 20:37:29.907373 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.907322    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x48bh-vgjns"]
Oct 30 20:37:29.964626 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.964600    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz"]
Oct 30 20:37:29.967661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.967643    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz"
Oct 30 20:37:29.975010 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.974991    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"]
Oct 30 20:37:29.978015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.977999    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:29.992057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:29.992040    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz"]
Oct 30 20:37:30.000653 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.000638    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ckj7t\""
Oct 30 20:37:30.001458 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.001441    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 20:37:30.006987 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.006958    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f77f82-84fd-496c-994e-7fdc2a018af3-tmp\") pod \"downloads-6dd789c8b7-g57td\" (UID: \"d9f77f82-84fd-496c-994e-7fdc2a018af3\") " pod="openshift-console/downloads-6dd789c8b7-g57td"
Oct 30 20:37:30.007073 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.007001    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96blw\" (UniqueName: \"kubernetes.io/projected/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-kube-api-access-96blw\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.007122 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.007087    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-trusted-ca-bundle\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.007155 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.007142    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-service-ca-bundle\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.007229 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.007216    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9ltt\" (UniqueName: \"kubernetes.io/projected/d9f77f82-84fd-496c-994e-7fdc2a018af3-kube-api-access-t9ltt\") pod \"downloads-6dd789c8b7-g57td\" (UID: \"d9f77f82-84fd-496c-994e-7fdc2a018af3\") " pod="openshift-console/downloads-6dd789c8b7-g57td"
Oct 30 20:37:30.007313 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.007294    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-archives-path\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.108307 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108277    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-service-ca-bundle\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.108307 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108309    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fwm8\" (UniqueName: \"kubernetes.io/projected/baa5eb14-051a-4be5-be3a-bc23b427540e-kube-api-access-5fwm8\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.108497 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108333    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/60a8f033-c381-47bd-aa2a-bf531bc18359-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-ssrdz\" (UID: \"60a8f033-c381-47bd-aa2a-bf531bc18359\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz"
Oct 30 20:37:30.108497 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108362    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/baa5eb14-051a-4be5-be3a-bc23b427540e-image-registry-private-configuration\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.108497 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108446    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/baa5eb14-051a-4be5-be3a-bc23b427540e-bound-sa-token\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.108497 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108488    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/baa5eb14-051a-4be5-be3a-bc23b427540e-ca-trust-extracted\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.108692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108521    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9ltt\" (UniqueName: \"kubernetes.io/projected/d9f77f82-84fd-496c-994e-7fdc2a018af3-kube-api-access-t9ltt\") pod \"downloads-6dd789c8b7-g57td\" (UID: \"d9f77f82-84fd-496c-994e-7fdc2a018af3\") " pod="openshift-console/downloads-6dd789c8b7-g57td"
Oct 30 20:37:30.108692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108571    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-archives-path\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.108692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108588    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/baa5eb14-051a-4be5-be3a-bc23b427540e-installation-pull-secrets\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.108692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108614    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/baa5eb14-051a-4be5-be3a-bc23b427540e-registry-tls\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.108692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108634    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f77f82-84fd-496c-994e-7fdc2a018af3-tmp\") pod \"downloads-6dd789c8b7-g57td\" (UID: \"d9f77f82-84fd-496c-994e-7fdc2a018af3\") " pod="openshift-console/downloads-6dd789c8b7-g57td"
Oct 30 20:37:30.108692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108655    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96blw\" (UniqueName: \"kubernetes.io/projected/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-kube-api-access-96blw\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.108692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108677    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-trusted-ca-bundle\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.109078 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108701    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/baa5eb14-051a-4be5-be3a-bc23b427540e-trusted-ca\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.109078 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.108777    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/baa5eb14-051a-4be5-be3a-bc23b427540e-registry-certificates\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.109295 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.109222    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-archives-path\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.109382 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.109359    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9f77f82-84fd-496c-994e-7fdc2a018af3-tmp\") pod \"downloads-6dd789c8b7-g57td\" (UID: \"d9f77f82-84fd-496c-994e-7fdc2a018af3\") " pod="openshift-console/downloads-6dd789c8b7-g57td"
Oct 30 20:37:30.109577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.109554    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-service-ca-bundle\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.109791 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.109772    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-trusted-ca-bundle\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.115438 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.115415    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"]
Oct 30 20:37:30.124297 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.124273    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:30.127608 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.127589    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96blw\" (UniqueName: \"kubernetes.io/projected/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-kube-api-access-96blw\") pod \"periodic-gathering-x48bh-vgjns\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") " pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.127700 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.127641    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9ltt\" (UniqueName: \"kubernetes.io/projected/d9f77f82-84fd-496c-994e-7fdc2a018af3-kube-api-access-t9ltt\") pod \"downloads-6dd789c8b7-g57td\" (UID: \"d9f77f82-84fd-496c-994e-7fdc2a018af3\") " pod="openshift-console/downloads-6dd789c8b7-g57td"
Oct 30 20:37:30.200479 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.200430    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-g57td"
Oct 30 20:37:30.208284 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.208269    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:37:30.209539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.209520    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/baa5eb14-051a-4be5-be3a-bc23b427540e-trusted-ca\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.209641 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.209548    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/baa5eb14-051a-4be5-be3a-bc23b427540e-registry-certificates\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.209641 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.209570    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fwm8\" (UniqueName: \"kubernetes.io/projected/baa5eb14-051a-4be5-be3a-bc23b427540e-kube-api-access-5fwm8\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.209641 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.209600    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/60a8f033-c381-47bd-aa2a-bf531bc18359-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-ssrdz\" (UID: \"60a8f033-c381-47bd-aa2a-bf531bc18359\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz"
Oct 30 20:37:30.209800 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.209642    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/baa5eb14-051a-4be5-be3a-bc23b427540e-image-registry-private-configuration\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.209879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.209821    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/baa5eb14-051a-4be5-be3a-bc23b427540e-bound-sa-token\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.210166 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.210141    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/baa5eb14-051a-4be5-be3a-bc23b427540e-ca-trust-extracted\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.210336 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.210315    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/baa5eb14-051a-4be5-be3a-bc23b427540e-installation-pull-secrets\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.210468 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.210451    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/baa5eb14-051a-4be5-be3a-bc23b427540e-registry-tls\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.210623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.210469    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/baa5eb14-051a-4be5-be3a-bc23b427540e-ca-trust-extracted\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.210719 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.210696    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/baa5eb14-051a-4be5-be3a-bc23b427540e-trusted-ca\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.210790 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.210768    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/baa5eb14-051a-4be5-be3a-bc23b427540e-registry-certificates\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.212694 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.212672    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/baa5eb14-051a-4be5-be3a-bc23b427540e-image-registry-private-configuration\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.213084 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.213059    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/60a8f033-c381-47bd-aa2a-bf531bc18359-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-ssrdz\" (UID: \"60a8f033-c381-47bd-aa2a-bf531bc18359\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz"
Oct 30 20:37:30.213388 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.213371    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/baa5eb14-051a-4be5-be3a-bc23b427540e-registry-tls\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.214234 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.214180    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/baa5eb14-051a-4be5-be3a-bc23b427540e-installation-pull-secrets\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.219265 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.219241    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/baa5eb14-051a-4be5-be3a-bc23b427540e-bound-sa-token\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.219986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.219963    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fwm8\" (UniqueName: \"kubernetes.io/projected/baa5eb14-051a-4be5-be3a-bc23b427540e-kube-api-access-5fwm8\") pod \"image-registry-56dfd5c8cd-9dzxh\" (UID: \"baa5eb14-051a-4be5-be3a-bc23b427540e\") " pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.238935 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.238890    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:30.279296 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.279267    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz"
Oct 30 20:37:30.288171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.288012    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.343635 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.343607    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:30.458611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.458532    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:30.484360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.484333    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-g57td"]
Oct 30 20:37:30.485407 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.485387    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x48bh-vgjns"]
Oct 30 20:37:30.487724 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:30.487693    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f77f82_84fd_496c_994e_7fdc2a018af3.slice/crio-06ac2167901e910d0e6cb82c95f0cddcdf18b651b085fc173cde088097b3ddac WatchSource:0}: Error finding container 06ac2167901e910d0e6cb82c95f0cddcdf18b651b085fc173cde088097b3ddac: Status 404 returned error can't find the container with id 06ac2167901e910d0e6cb82c95f0cddcdf18b651b085fc173cde088097b3ddac
Oct 30 20:37:30.488432 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:30.488364    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f567eec_9295_4e3e_bf6f_ec0e187eb3ed.slice/crio-be05cd365cd43efd11b6996141c40bc99ddd29342ba9f9aa7543396767fd386c WatchSource:0}: Error finding container be05cd365cd43efd11b6996141c40bc99ddd29342ba9f9aa7543396767fd386c: Status 404 returned error can't find the container with id be05cd365cd43efd11b6996141c40bc99ddd29342ba9f9aa7543396767fd386c
Oct 30 20:37:30.563710 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.563677    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz"]
Oct 30 20:37:30.564229 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.564185    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:30.566628 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.566605    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"]
Oct 30 20:37:30.567319 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:30.567296    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a8f033_c381_47bd_aa2a_bf531bc18359.slice/crio-5840f3097e615a62247d36b4d3bf53d85027728d3ef6c9b4f5d622e3aa2582a9 WatchSource:0}: Error finding container 5840f3097e615a62247d36b4d3bf53d85027728d3ef6c9b4f5d622e3aa2582a9: Status 404 returned error can't find the container with id 5840f3097e615a62247d36b4d3bf53d85027728d3ef6c9b4f5d622e3aa2582a9
Oct 30 20:37:30.569816 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:30.569792    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa5eb14_051a_4be5_be3a_bc23b427540e.slice/crio-3bdbb218b3a8ce183b9d4c230acd6365147fec06ec16063186aa499212f6e7fb WatchSource:0}: Error finding container 3bdbb218b3a8ce183b9d4c230acd6365147fec06ec16063186aa499212f6e7fb: Status 404 returned error can't find the container with id 3bdbb218b3a8ce183b9d4c230acd6365147fec06ec16063186aa499212f6e7fb
Oct 30 20:37:30.790329 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.790263    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nfvzn"
Oct 30 20:37:30.833601 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.833560    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x48bh-vgjns" event={"ID":"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed","Type":"ContainerStarted","Data":"87f7a8d97c0a0c3a0fccf8c2d968bdfb22862bb8d4e92f7227baa0f9f58c4adb"}
Oct 30 20:37:30.833769 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.833625    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x48bh-vgjns" event={"ID":"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed","Type":"ContainerStarted","Data":"be05cd365cd43efd11b6996141c40bc99ddd29342ba9f9aa7543396767fd386c"}
Oct 30 20:37:30.834918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.834892    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-g57td" event={"ID":"d9f77f82-84fd-496c-994e-7fdc2a018af3","Type":"ContainerStarted","Data":"06ac2167901e910d0e6cb82c95f0cddcdf18b651b085fc173cde088097b3ddac"}
Oct 30 20:37:30.836337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.836316    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz" event={"ID":"60a8f033-c381-47bd-aa2a-bf531bc18359","Type":"ContainerStarted","Data":"5840f3097e615a62247d36b4d3bf53d85027728d3ef6c9b4f5d622e3aa2582a9"}
Oct 30 20:37:30.838154 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.838134    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh" event={"ID":"baa5eb14-051a-4be5-be3a-bc23b427540e","Type":"ContainerStarted","Data":"875830919d039cbb5a8c314d78515805cf6f33f3a3616a1b8c48271d4c48e19b"}
Oct 30 20:37:30.838261 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.838168    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh" event={"ID":"baa5eb14-051a-4be5-be3a-bc23b427540e","Type":"ContainerStarted","Data":"3bdbb218b3a8ce183b9d4c230acd6365147fec06ec16063186aa499212f6e7fb"}
Oct 30 20:37:30.838261 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.838226    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:30.849427 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.849379    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-x48bh-vgjns" podStartSLOduration=1.849368797 podStartE2EDuration="1.849368797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:30.848940861 +0000 UTC m=+17.340757406" watchObservedRunningTime="2026-10-30 20:37:30.849368797 +0000 UTC m=+17.341185341"
Oct 30 20:37:30.868259 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:30.868211    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh" podStartSLOduration=1.8681943250000002 podStartE2EDuration="1.868194325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:30.867433283 +0000 UTC m=+17.359249828" watchObservedRunningTime="2026-10-30 20:37:30.868194325 +0000 UTC m=+17.360010873"
Oct 30 20:37:31.314988 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:31.314960    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nfvzn_6140cde0-b52b-48e1-a1d6-bcac35086922/dns/0.log"
Oct 30 20:37:31.514454 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:31.514429    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nfvzn_6140cde0-b52b-48e1-a1d6-bcac35086922/kube-rbac-proxy/0.log"
Oct 30 20:37:31.720796 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:31.720774    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4mv46_7389168a-7f19-48df-98f8-6d6d20892416/dns-node-resolver/0.log"
Oct 30 20:37:31.845833 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:31.845785    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz" event={"ID":"60a8f033-c381-47bd-aa2a-bf531bc18359","Type":"ContainerStarted","Data":"04b61583815d643daea7ffb4f003a0cf27dde47a53e52da218793390529032e4"}
Oct 30 20:37:31.863182 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:31.863132    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz" podStartSLOduration=1.7850075570000001 podStartE2EDuration="2.863112998s" totalImagesPullingTime="1.078105441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:31.862804483 +0000 UTC m=+18.354621027" watchObservedRunningTime="2026-10-30 20:37:31.863112998 +0000 UTC m=+18.354929571"
Oct 30 20:37:31.915278 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:31.915201    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56dfd5c8cd-9dzxh_baa5eb14-051a-4be5-be3a-bc23b427540e/registry/0.log"
Oct 30 20:37:32.115585 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:32.115556    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bfdcc7f98-zlptz_204ba748-a514-4772-9fba-dfafd3eddcf7/registry/0.log"
Oct 30 20:37:32.315786 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:32.315707    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-948b95b59-tdzdl_4dd510fb-f40e-401c-845f-23b9d750dc68/registry/0.log"
Oct 30 20:37:32.515230 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:32.515198    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sq5lx_58b187ea-dead-4a37-8a52-5e447783d815/node-ca/0.log"
Oct 30 20:37:32.715392 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:32.715362    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dc99f8759-pzknq_6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/router/0.log"
Oct 30 20:37:32.858164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:32.858132    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz"
Oct 30 20:37:32.863728 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:32.863694    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-ssrdz"
Oct 30 20:37:32.914395 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:32.914342    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9v7g_93f061fe-5c27-419b-8677-9734d5dce379/serve-healthcheck-canary/0.log"
Oct 30 20:37:33.080668 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.080582    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:33.114690 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.114657    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"]
Oct 30 20:37:33.118572 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.118548    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.127990 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.127949    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 20:37:33.127990 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.127971    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 20:37:33.128147 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.128018    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:37:33.128147 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.128088    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b5b2f\""
Oct 30 20:37:33.129064 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.129041    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"]
Oct 30 20:37:33.157908 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.157879    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:33.235544 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.235504    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:33.239502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.239473    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.239612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.239515    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.239725 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.239689    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.239850 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.239788    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpcdp\" (UniqueName: \"kubernetes.io/projected/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-kube-api-access-vpcdp\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.341190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.341117    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.341190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.341154    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.341672 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.341226    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.341672 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.341250    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpcdp\" (UniqueName: \"kubernetes.io/projected/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-kube-api-access-vpcdp\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.342115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.342075    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.343918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.343893    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.344137 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.344115    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.350436 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.350415    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpcdp\" (UniqueName: \"kubernetes.io/projected/11c30c97-5275-4fc9-ab81-54c0f3de3ec2-kube-api-access-vpcdp\") pod \"prometheus-operator-9cd89b74c-2fxhd\" (UID: \"11c30c97-5275-4fc9-ab81-54c0f3de3ec2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.436012 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.435974    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"
Oct 30 20:37:33.443716 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.443668    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:33.546959 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.546920    4810 patch_prober.go:29] interesting pod/image-registry-948b95b59-tdzdl container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:37:33.547072 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.546967    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-948b95b59-tdzdl" podUID="4dd510fb-f40e-401c-845f-23b9d750dc68" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:37:33.553570 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.553486    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:33.640820 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.640771    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:33.755223 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.755191    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd"]
Oct 30 20:37:33.759625 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:33.759595    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c30c97_5275_4fc9_ab81_54c0f3de3ec2.slice/crio-7492ca7f957be464767cb3b093504a2d25bf2cb923644c316264828e721040f4 WatchSource:0}: Error finding container 7492ca7f957be464767cb3b093504a2d25bf2cb923644c316264828e721040f4: Status 404 returned error can't find the container with id 7492ca7f957be464767cb3b093504a2d25bf2cb923644c316264828e721040f4
Oct 30 20:37:33.772687 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.772610    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:33.869485 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.869448    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd" event={"ID":"11c30c97-5275-4fc9-ab81-54c0f3de3ec2","Type":"ContainerStarted","Data":"7492ca7f957be464767cb3b093504a2d25bf2cb923644c316264828e721040f4"}
Oct 30 20:37:33.870530 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.870480    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:33.972156 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:33.972044    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:34.348786 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:34.348709    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:34.559465 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:34.559429    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:34.655972 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:34.655935    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:34.885662 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:34.885624    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:34.961031 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:34.960954    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:35.287206 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:35.287118    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:35.387141 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:35.387107    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:35.660297 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:35.660263    4810 patch_prober.go:29] interesting pod/image-registry-948b95b59-tdzdl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:37:35.660462 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:35.660317    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-948b95b59-tdzdl" podUID="4dd510fb-f40e-401c-845f-23b9d750dc68" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:37:35.882334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:35.882290    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd" event={"ID":"11c30c97-5275-4fc9-ab81-54c0f3de3ec2","Type":"ContainerStarted","Data":"f510000a0beefe00f2529a361bcaa1b3d06dd768d06e41cff20ec2c8a1410b0a"}
Oct 30 20:37:35.882501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:35.882354    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd" event={"ID":"11c30c97-5275-4fc9-ab81-54c0f3de3ec2","Type":"ContainerStarted","Data":"50dea0dd2d61018d5395b3c90ac671c1b7cbdc83a3abd45b86c37c20fa947132"}
Oct 30 20:37:35.903251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:35.903194    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2fxhd" podStartSLOduration=1.6750239630000001 podStartE2EDuration="2.903176522s" totalImagesPullingTime="1.228152559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:35.901849752 +0000 UTC m=+22.393666298" watchObservedRunningTime="2026-10-30 20:37:35.903176522 +0000 UTC m=+22.394993069"
Oct 30 20:37:36.281573 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:36.281533    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:36.355283 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:36.355246    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:36.421470 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:36.421444    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4spg"
Oct 30 20:37:36.687600 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:36.687564    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:36.784634 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:36.784603    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:37.334768 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.334738    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:37.472095 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.472056    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:37.519000 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.518963    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"]
Oct 30 20:37:37.524581 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.524554    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.530329 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.530307    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"]
Oct 30 20:37:37.541294 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.534845    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.546251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.546228    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 20:37:37.546378 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.546305    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:37:37.546378 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.546355    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:37:37.546504 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.546434    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rthwp\""
Oct 30 20:37:37.546564 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.546511    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 20:37:37.546564 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.546545    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-857c4b849c-pg2d4"]
Oct 30 20:37:37.550078 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.550059    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"]
Oct 30 20:37:37.550246 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.550143    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.551940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.551775    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2wdwm\""
Oct 30 20:37:37.553434 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.553378    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 20:37:37.556827 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.556355    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tmxfc"]
Oct 30 20:37:37.560250 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.560230    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"]
Oct 30 20:37:37.560361 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.560274    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857c4b849c-pg2d4"]
Oct 30 20:37:37.560425 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.560372    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.592102 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.592042    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ppxb4\""
Oct 30 20:37:37.592233 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.592164    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:37:37.594005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.593935    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:37:37.594005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.593989    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:37.594571 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.594539    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 20:37:37.594987 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.594885    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 20:37:37.599094 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.599073    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn4xz\" (UniqueName: \"kubernetes.io/projected/e3e46551-28b8-47a0-ad33-59151c3137d2-kube-api-access-wn4xz\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.599204 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.599105    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.599204 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.599132    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.599318 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.599201    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.599318 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.599307    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5c74x\""
Oct 30 20:37:37.599416 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.599349    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-textfile\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.599472 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.599432    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3e46551-28b8-47a0-ad33-59151c3137d2-metrics-client-ca\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.599472 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.599465    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.600001 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.599975    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.600068 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600014    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-serving-cert\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.600068 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600036    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4xz\" (UniqueName: \"kubernetes.io/projected/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-kube-api-access-4j4xz\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.600068 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600063    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg546\" (UniqueName: \"kubernetes.io/projected/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-kube-api-access-vg546\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.600214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600091    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-service-ca\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.600214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600112    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-wtmp\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.600214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600136    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.600214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600162    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-tmp\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.600214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600198    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.600461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600224    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdjh9\" (UniqueName: \"kubernetes.io/projected/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-api-access-jdjh9\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.600461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600246    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3e46551-28b8-47a0-ad33-59151c3137d2-root\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.600461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600266    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.600461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600300    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-config\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.600461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600322    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.600461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600345    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3e46551-28b8-47a0-ad33-59151c3137d2-sys\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.600461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600366    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-accelerators-collector-config\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.600461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600404    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-oauth-config\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.600461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600438    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-oauth-serving-cert\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.600461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.600459    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-tls\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.619945 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.619831    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 20:37:37.619945 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.619899    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 20:37:37.620316 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.620041    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:37:37.620316 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.620115    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 20:37:37.684157 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.684121    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:37.701116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701086    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3e46551-28b8-47a0-ad33-59151c3137d2-sys\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701253 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701128    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-accelerators-collector-config\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701253 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701201    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3e46551-28b8-47a0-ad33-59151c3137d2-sys\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701253 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701241    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-oauth-config\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.701422 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701288    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-oauth-serving-cert\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.701422 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701316    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-tls\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701422 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701352    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn4xz\" (UniqueName: \"kubernetes.io/projected/e3e46551-28b8-47a0-ad33-59151c3137d2-kube-api-access-wn4xz\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701422 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701377    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.701620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701424    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.701620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701461    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.701620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701500    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-textfile\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701525    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3e46551-28b8-47a0-ad33-59151c3137d2-metrics-client-ca\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701550    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701590    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.701620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701616    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-serving-cert\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701638    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4xz\" (UniqueName: \"kubernetes.io/projected/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-kube-api-access-4j4xz\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701669    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg546\" (UniqueName: \"kubernetes.io/projected/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-kube-api-access-vg546\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701706    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-service-ca\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701731    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-wtmp\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701736    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-accelerators-collector-config\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701763    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701791    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-tmp\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701836    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701885    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdjh9\" (UniqueName: \"kubernetes.io/projected/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-api-access-jdjh9\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701912    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3e46551-28b8-47a0-ad33-59151c3137d2-root\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701938    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.701996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.701983    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-config\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.702588 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.702010    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.702588 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.702144    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.702588 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.702185    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-oauth-serving-cert\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.702588 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.702241    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3e46551-28b8-47a0-ad33-59151c3137d2-metrics-client-ca\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.702588 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.702296    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-wtmp\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.702870 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.702836    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-textfile\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.702942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.702927    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-tmp\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.703524 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.703178    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.703524 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.703259    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3e46551-28b8-47a0-ad33-59151c3137d2-root\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.703677 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.703619    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.704234 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.703934    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-config\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.705339 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.705314    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.705722 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.705697    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-service-ca\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.705893 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.705852    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.706124 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.706100    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.706206 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.706178    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-tls\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.706569 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.706540    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.707504 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.707481    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3e46551-28b8-47a0-ad33-59151c3137d2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.707575 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.707535    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-oauth-config\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.708532 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.708277    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.709573 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.709551    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-serving-cert\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.713731 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.713708    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn4xz\" (UniqueName: \"kubernetes.io/projected/e3e46551-28b8-47a0-ad33-59151c3137d2-kube-api-access-wn4xz\") pod \"node-exporter-tmxfc\" (UID: \"e3e46551-28b8-47a0-ad33-59151c3137d2\") " pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.714177 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.714155    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg546\" (UniqueName: \"kubernetes.io/projected/a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54-kube-api-access-vg546\") pod \"openshift-state-metrics-7bd7489957-qtvts\" (UID: \"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.714740 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.714720    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdjh9\" (UniqueName: \"kubernetes.io/projected/c30e377f-0ccf-4b99-af6e-2711bf8f7bdb-kube-api-access-jdjh9\") pod \"kube-state-metrics-5dc8d7d89b-h9kdb\" (UID: \"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.714980 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.714962    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4xz\" (UniqueName: \"kubernetes.io/projected/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-kube-api-access-4j4xz\") pod \"console-857c4b849c-pg2d4\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") " pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.769144 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.769112    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:37.857027 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.856951    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"
Oct 30 20:37:37.863828 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.863803    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:37.873026 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.872972    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"
Oct 30 20:37:37.877797 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.877611    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:37.909539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.909509    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tmxfc"
Oct 30 20:37:37.982755 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:37.982682    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:38.077049 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.077015    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:38.166156 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.166129    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts"]
Oct 30 20:37:38.168922 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.168894    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb"]
Oct 30 20:37:38.170260 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:38.170090    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52fc8a5_ace2_44a9_a2b1_7d4a9d2d4d54.slice/crio-8449098a05e50b854f21c552f14a64c83f0a38aed1dfcf750fdfaa21c220df9e WatchSource:0}: Error finding container 8449098a05e50b854f21c552f14a64c83f0a38aed1dfcf750fdfaa21c220df9e: Status 404 returned error can't find the container with id 8449098a05e50b854f21c552f14a64c83f0a38aed1dfcf750fdfaa21c220df9e
Oct 30 20:37:38.170918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.170892    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:38.173509 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:38.173209    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30e377f_0ccf_4b99_af6e_2711bf8f7bdb.slice/crio-113761d90f89ed32cad17418c5161a732daf793022f2a2ae4932e9d8fc6ae929 WatchSource:0}: Error finding container 113761d90f89ed32cad17418c5161a732daf793022f2a2ae4932e9d8fc6ae929: Status 404 returned error can't find the container with id 113761d90f89ed32cad17418c5161a732daf793022f2a2ae4932e9d8fc6ae929
Oct 30 20:37:38.195423 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.195330    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857c4b849c-pg2d4"]
Oct 30 20:37:38.200953 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:38.200927    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef43ee01_2c9f_48e4_a1ac_9f21dc308579.slice/crio-ccb8a686c2c660e86f6e72f6552c7ee377b01f8c818f6f42c22d2379f975a6f0 WatchSource:0}: Error finding container ccb8a686c2c660e86f6e72f6552c7ee377b01f8c818f6f42c22d2379f975a6f0: Status 404 returned error can't find the container with id ccb8a686c2c660e86f6e72f6552c7ee377b01f8c818f6f42c22d2379f975a6f0
Oct 30 20:37:38.518412 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.518188    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:38.626450 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.626417    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:37:38.633569 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.631466    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.635295 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.635232    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:38.641702 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.641557    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 20:37:38.644123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.642359    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 20:37:38.644123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.642404    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 20:37:38.644123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.642482    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-92xsf\""
Oct 30 20:37:38.644123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.642532    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 20:37:38.644123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.642570    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 20:37:38.644123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.642601    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 20:37:38.644123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.642623    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 20:37:38.644123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.642653    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 20:37:38.676546 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.676446    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:37:38.691156 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.691056    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.716950    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717039    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717065    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-config-out\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717099    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717134    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpbgm\" (UniqueName: \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-kube-api-access-cpbgm\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717165    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717192    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717217    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717240    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717268    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-web-config\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717297    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717361    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.717611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.717386    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.765824 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.765772    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:38.819475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.818481    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.819475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.818529    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-config-out\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.819475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.818568    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.819475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.818626    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbgm\" (UniqueName: \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-kube-api-access-cpbgm\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.819475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.818654    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.819475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.818705    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.819475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.818733    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.819475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.818778    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.819475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.818814    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-web-config\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.819475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.818843    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.820138 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.819702    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.821915 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.820297    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.821915 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.820335    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.821915 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.820382    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.821915 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.821165    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.821915 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:38.821282    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle podName:8a82d659-a274-4c7b-9935-5004f715b456 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:39.321262253 +0000 UTC m=+25.813078781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "8a82d659-a274-4c7b-9935-5004f715b456") : configmap references non-existent config key: ca-bundle.crt
Oct 30 20:37:38.827845 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.827513    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.828908 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.828425    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.829468 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.829447    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-web-config\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.830086 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.830042    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.835032 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.834989    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.840118 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.840040    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.840796 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.840702    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.841434 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.841339    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.842469 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.841784    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-config-out\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.845725 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.845600    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbgm\" (UniqueName: \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-kube-api-access-cpbgm\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:38.899601 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.899563    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tmxfc" event={"ID":"e3e46551-28b8-47a0-ad33-59151c3137d2","Type":"ContainerStarted","Data":"82e540d0ae74a58b90ef3462fef4133878fbd0b088a17b64421fc91bde0c2587"}
Oct 30 20:37:38.901027 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.900970    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c4b849c-pg2d4" event={"ID":"ef43ee01-2c9f-48e4-a1ac-9f21dc308579","Type":"ContainerStarted","Data":"ccb8a686c2c660e86f6e72f6552c7ee377b01f8c818f6f42c22d2379f975a6f0"}
Oct 30 20:37:38.902505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.902479    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb" event={"ID":"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb","Type":"ContainerStarted","Data":"113761d90f89ed32cad17418c5161a732daf793022f2a2ae4932e9d8fc6ae929"}
Oct 30 20:37:38.905037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.904923    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts" event={"ID":"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54","Type":"ContainerStarted","Data":"acbbcb3d4754b1135193e95314e253faf05e418cbe79c00f687ee464956915bf"}
Oct 30 20:37:38.905037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.904976    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts" event={"ID":"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54","Type":"ContainerStarted","Data":"ae34ad88ede2bf2d60e88e3ae535298c03b62eeb56b099f3f1dd1b41801cd35b"}
Oct 30 20:37:38.905037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:38.905003    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts" event={"ID":"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54","Type":"ContainerStarted","Data":"8449098a05e50b854f21c552f14a64c83f0a38aed1dfcf750fdfaa21c220df9e"}
Oct 30 20:37:39.324957 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.324228    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:39.324957 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:39.324498    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle podName:8a82d659-a274-4c7b-9935-5004f715b456 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:40.324475489 +0000 UTC m=+26.816292019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "8a82d659-a274-4c7b-9935-5004f715b456") : configmap references non-existent config key: ca-bundle.crt
Oct 30 20:37:39.604557 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.604480    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-744b9bc9d7-grkth"]
Oct 30 20:37:39.619098 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.619068    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.624010 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.623938    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-744b9bc9d7-grkth"]
Oct 30 20:37:39.629501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.628969    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 20:37:39.629501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.629069    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 20:37:39.629501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.629106    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 20:37:39.629501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.629184    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-912o9imlerlne\""
Oct 30 20:37:39.629501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.629304    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x5vqj\""
Oct 30 20:37:39.629501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.629392    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 20:37:39.631584 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.631546    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 20:37:39.734263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.733719    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.734263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.733770    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.734263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.733797    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af70ce71-5273-4ef0-908e-c374184d46a2-tmp\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.734263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.733866    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.734263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.733901    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-grpc-tls\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.734263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.733938    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-tls\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.734263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.733967    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.734263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.733991    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q57ws\" (UniqueName: \"kubernetes.io/projected/af70ce71-5273-4ef0-908e-c374184d46a2-kube-api-access-q57ws\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.734263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.734051    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af70ce71-5273-4ef0-908e-c374184d46a2-metrics-client-ca\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.835601 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.835567    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.835763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.835618    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-grpc-tls\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.835763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.835658    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-tls\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.835763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.835689    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.835763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.835714    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q57ws\" (UniqueName: \"kubernetes.io/projected/af70ce71-5273-4ef0-908e-c374184d46a2-kube-api-access-q57ws\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.836020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.835775    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af70ce71-5273-4ef0-908e-c374184d46a2-metrics-client-ca\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.836020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.835911    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.836020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.835941    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.836020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.835965    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af70ce71-5273-4ef0-908e-c374184d46a2-tmp\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.836286 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.836261    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af70ce71-5273-4ef0-908e-c374184d46a2-tmp\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.836760 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.836705    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af70ce71-5273-4ef0-908e-c374184d46a2-metrics-client-ca\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.838772 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.838745    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.839167 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.839142    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.845695 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.845670    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q57ws\" (UniqueName: \"kubernetes.io/projected/af70ce71-5273-4ef0-908e-c374184d46a2-kube-api-access-q57ws\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.847444 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.847419    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.847547 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.847508    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.848070 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.848029    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-thanos-querier-tls\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.850726 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.850326    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/af70ce71-5273-4ef0-908e-c374184d46a2-secret-grpc-tls\") pod \"thanos-querier-744b9bc9d7-grkth\" (UID: \"af70ce71-5273-4ef0-908e-c374184d46a2\") " pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:39.903263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.903063    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:39.913490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.913408    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tmxfc" event={"ID":"e3e46551-28b8-47a0-ad33-59151c3137d2","Type":"ContainerStarted","Data":"d17e8e298a9f1cb3f135f90d8db47862770b8ca277fd60c6dffa44e65a19aee2"}
Oct 30 20:37:39.964419 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:39.964386    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:40.045798 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.045747    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:40.059235 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.056791    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:40.341834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.341749    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:40.342967 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.342937    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:40.366401 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.366374    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-744b9bc9d7-grkth"]
Oct 30 20:37:40.369380 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:40.369351    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf70ce71_5273_4ef0_908e_c374184d46a2.slice/crio-e2475a02b34aabeceb116dd378a96dbac71ea010528e2c8ada8addf9b5deb916 WatchSource:0}: Error finding container e2475a02b34aabeceb116dd378a96dbac71ea010528e2c8ada8addf9b5deb916: Status 404 returned error can't find the container with id e2475a02b34aabeceb116dd378a96dbac71ea010528e2c8ada8addf9b5deb916
Oct 30 20:37:40.476104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.475884    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:37:40.613041 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.612961    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:40.719288 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.719140    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:40.805225 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.805195    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:40.929423 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.928889    4810 generic.go:356] "Generic (PLEG): container finished" podID="e3e46551-28b8-47a0-ad33-59151c3137d2" containerID="d17e8e298a9f1cb3f135f90d8db47862770b8ca277fd60c6dffa44e65a19aee2" exitCode=0
Oct 30 20:37:40.929423 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.929015    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tmxfc" event={"ID":"e3e46551-28b8-47a0-ad33-59151c3137d2","Type":"ContainerDied","Data":"d17e8e298a9f1cb3f135f90d8db47862770b8ca277fd60c6dffa44e65a19aee2"}
Oct 30 20:37:40.934888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.933552    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb" event={"ID":"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb","Type":"ContainerStarted","Data":"0d680eea02710cbdb25fddb087e07bb6faf5703d1aecbdddcaf506ae8f656678"}
Oct 30 20:37:40.934888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.933668    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb" event={"ID":"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb","Type":"ContainerStarted","Data":"b41189ce538430457506bb5b155f457f9b4214bb551e52a907a1600ca78aaa6f"}
Oct 30 20:37:40.942416 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.941686    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts" event={"ID":"a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54","Type":"ContainerStarted","Data":"0040defbc5fc83d420112e3374a2acba807cac0e3e8eea1be9775b6f5b4d7d37"}
Oct 30 20:37:40.944706 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.944612    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth" event={"ID":"af70ce71-5273-4ef0-908e-c374184d46a2","Type":"ContainerStarted","Data":"e2475a02b34aabeceb116dd378a96dbac71ea010528e2c8ada8addf9b5deb916"}
Oct 30 20:37:40.964459 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.964340    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:40.977299 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.973225    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qtvts" podStartSLOduration=2.275055733 podStartE2EDuration="3.973204001s" totalImagesPullingTime="1.698148268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:40.972636335 +0000 UTC m=+27.464452880" watchObservedRunningTime="2026-10-30 20:37:40.973204001 +0000 UTC m=+27.465020560"
Oct 30 20:37:40.990945 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:40.980507    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:37:40.990945 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:40.990665    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a82d659_a274_4c7b_9935_5004f715b456.slice/crio-b0443d9e3499471e18cb268189df6499d83fdee019fafd73002c4c2b5c1bc776 WatchSource:0}: Error finding container b0443d9e3499471e18cb268189df6499d83fdee019fafd73002c4c2b5c1bc776: Status 404 returned error can't find the container with id b0443d9e3499471e18cb268189df6499d83fdee019fafd73002c4c2b5c1bc776
Oct 30 20:37:41.667352 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:41.667316    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:41.740232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:41.740199    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:41.956769 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:41.955488    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb" event={"ID":"c30e377f-0ccf-4b99-af6e-2711bf8f7bdb","Type":"ContainerStarted","Data":"daafc94b28d50b03081c33fbe5029c5f3fe4f5eaeeefc455af9958a63ff77931"}
Oct 30 20:37:41.958152 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:41.958043    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerStarted","Data":"b0443d9e3499471e18cb268189df6499d83fdee019fafd73002c4c2b5c1bc776"}
Oct 30 20:37:41.960917 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:41.960759    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:41.972828 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:41.971541    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tmxfc" event={"ID":"e3e46551-28b8-47a0-ad33-59151c3137d2","Type":"ContainerStarted","Data":"d210c6f4f4563a783120bd18965b150a488e5605b866bc0721c8739d9ad1c08d"}
Oct 30 20:37:41.972828 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:41.972145    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tmxfc" event={"ID":"e3e46551-28b8-47a0-ad33-59151c3137d2","Type":"ContainerStarted","Data":"08502c9a45d92bb6ee732a000ccbe64ca07b9b10eb261963cb859997ea040a69"}
Oct 30 20:37:41.985852 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:41.985774    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h9kdb" podStartSLOduration=2.481470646 podStartE2EDuration="4.985736069s" totalImagesPullingTime="2.504265423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:41.984419633 +0000 UTC m=+28.476236181" watchObservedRunningTime="2026-10-30 20:37:41.985736069 +0000 UTC m=+28.477552619"
Oct 30 20:37:42.017287 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.016806    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tmxfc" podStartSLOduration=3.22030535 podStartE2EDuration="5.016783879s" totalImagesPullingTime="1.70646057s" totalInitContainerRuntime="90.017959ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:42.014555219 +0000 UTC m=+28.506371768" watchObservedRunningTime="2026-10-30 20:37:42.016783879 +0000 UTC m=+28.508600424"
Oct 30 20:37:42.047114 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.047081    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f8657c7b9-5v8td"]
Oct 30 20:37:42.061332 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.061196    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:42.079068 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.079006    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f8657c7b9-5v8td"]
Oct 30 20:37:42.079225 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.079158    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.086097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.085581    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 20:37:42.086097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.085680    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5tfxl\""
Oct 30 20:37:42.086097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.085720    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 20:37:42.086691 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.086550    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 20:37:42.086691 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.086633    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 20:37:42.088180 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.088020    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fctrk3mdqobd3\""
Oct 30 20:37:42.170895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.170354    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05f34d74-7115-42fb-a288-27bf8ef7c3e8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.170895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.170428    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/05f34d74-7115-42fb-a288-27bf8ef7c3e8-secret-metrics-server-client-certs\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.170895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.170485    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/05f34d74-7115-42fb-a288-27bf8ef7c3e8-client-ca-bundle\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.170895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.170520    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq6c7\" (UniqueName: \"kubernetes.io/projected/05f34d74-7115-42fb-a288-27bf8ef7c3e8-kube-api-access-tq6c7\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.170895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.170556    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/05f34d74-7115-42fb-a288-27bf8ef7c3e8-secret-metrics-server-tls\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.170895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.170589    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/05f34d74-7115-42fb-a288-27bf8ef7c3e8-audit-log\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.170895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.170643    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/05f34d74-7115-42fb-a288-27bf8ef7c3e8-metrics-server-audit-profiles\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.273319 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.272472    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/05f34d74-7115-42fb-a288-27bf8ef7c3e8-secret-metrics-server-tls\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.273319 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.272530    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/05f34d74-7115-42fb-a288-27bf8ef7c3e8-audit-log\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.273319 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.272583    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/05f34d74-7115-42fb-a288-27bf8ef7c3e8-metrics-server-audit-profiles\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.273319 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.272674    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05f34d74-7115-42fb-a288-27bf8ef7c3e8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.273319 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.272724    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/05f34d74-7115-42fb-a288-27bf8ef7c3e8-secret-metrics-server-client-certs\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.273319 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.272773    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/05f34d74-7115-42fb-a288-27bf8ef7c3e8-client-ca-bundle\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.273319 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.272803    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq6c7\" (UniqueName: \"kubernetes.io/projected/05f34d74-7115-42fb-a288-27bf8ef7c3e8-kube-api-access-tq6c7\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.273319 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.273254    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/05f34d74-7115-42fb-a288-27bf8ef7c3e8-audit-log\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.276004 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.275925    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05f34d74-7115-42fb-a288-27bf8ef7c3e8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.276247 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.276217    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/05f34d74-7115-42fb-a288-27bf8ef7c3e8-metrics-server-audit-profiles\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.281206 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.278111    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/05f34d74-7115-42fb-a288-27bf8ef7c3e8-client-ca-bundle\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.283030 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.282305    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq6c7\" (UniqueName: \"kubernetes.io/projected/05f34d74-7115-42fb-a288-27bf8ef7c3e8-kube-api-access-tq6c7\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.283789 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.283752    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/05f34d74-7115-42fb-a288-27bf8ef7c3e8-secret-metrics-server-tls\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.292160 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.292134    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/05f34d74-7115-42fb-a288-27bf8ef7c3e8-secret-metrics-server-client-certs\") pod \"metrics-server-5f8657c7b9-5v8td\" (UID: \"05f34d74-7115-42fb-a288-27bf8ef7c3e8\") " pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.316831 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.316502    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"]
Oct 30 20:37:42.320297 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.320262    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:42.351882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.351675    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"]
Oct 30 20:37:42.352827 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.351805    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"
Oct 30 20:37:42.357217 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.357194    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 20:37:42.357343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.357285    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cr9vc\""
Oct 30 20:37:42.401979 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.401913    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:42.438367 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.438315    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:37:42.475756 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.475714    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07c3a947-1296-4e31-9d93-4fdb854fc889-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-84kx5\" (UID: \"07c3a947-1296-4e31-9d93-4fdb854fc889\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"
Oct 30 20:37:42.475913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.475882    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9bnz\" (UniqueName: \"kubernetes.io/projected/07c3a947-1296-4e31-9d93-4fdb854fc889-kube-api-access-q9bnz\") pod \"monitoring-plugin-77f4f76594-84kx5\" (UID: \"07c3a947-1296-4e31-9d93-4fdb854fc889\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"
Oct 30 20:37:42.502107 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.502043    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:42.578734 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.577781    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07c3a947-1296-4e31-9d93-4fdb854fc889-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-84kx5\" (UID: \"07c3a947-1296-4e31-9d93-4fdb854fc889\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"
Oct 30 20:37:42.578734 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.577914    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9bnz\" (UniqueName: \"kubernetes.io/projected/07c3a947-1296-4e31-9d93-4fdb854fc889-kube-api-access-q9bnz\") pod \"monitoring-plugin-77f4f76594-84kx5\" (UID: \"07c3a947-1296-4e31-9d93-4fdb854fc889\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"
Oct 30 20:37:42.584136 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.583455    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07c3a947-1296-4e31-9d93-4fdb854fc889-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-84kx5\" (UID: \"07c3a947-1296-4e31-9d93-4fdb854fc889\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"
Oct 30 20:37:42.589607 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.588469    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9bnz\" (UniqueName: \"kubernetes.io/projected/07c3a947-1296-4e31-9d93-4fdb854fc889-kube-api-access-q9bnz\") pod \"monitoring-plugin-77f4f76594-84kx5\" (UID: \"07c3a947-1296-4e31-9d93-4fdb854fc889\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"
Oct 30 20:37:42.602169 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.602087    4810 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:42.700159 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.700122    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"
Oct 30 20:37:42.736518 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.735481    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"]
Oct 30 20:37:42.762708 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.762674    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"]
Oct 30 20:37:42.762909 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.762813    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.767562 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.766631    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f8657c7b9-5v8td"]
Oct 30 20:37:42.770477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.770292    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 20:37:42.770477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.770362    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 20:37:42.770477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.770452    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 20:37:42.770771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.770483    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-h8rrn\""
Oct 30 20:37:42.770771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.770570    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 20:37:42.772636 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.771266    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 20:37:42.773721 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.773486    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 20:37:42.880626 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.880580    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa947095-7d5c-4e06-9e46-d297fe162fd3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.880795 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.880682    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-secret-telemeter-client\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.880795 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.880727    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.880795 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.880764    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-telemeter-client-tls\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.880996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.880804    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa947095-7d5c-4e06-9e46-d297fe162fd3-serving-certs-ca-bundle\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.880996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.880819    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-federate-client-tls\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.880996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.880882    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa947095-7d5c-4e06-9e46-d297fe162fd3-metrics-client-ca\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.880996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.880935    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klv4r\" (UniqueName: \"kubernetes.io/projected/aa947095-7d5c-4e06-9e46-d297fe162fd3-kube-api-access-klv4r\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.977417 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.977371    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td" event={"ID":"05f34d74-7115-42fb-a288-27bf8ef7c3e8","Type":"ContainerStarted","Data":"1be4ff119417ffaff751c087f3cce78a8a253b9e31c143b02bbab811bc59b992"}
Oct 30 20:37:42.981748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.981711    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa947095-7d5c-4e06-9e46-d297fe162fd3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.981912 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.981808    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-secret-telemeter-client\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.981912 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.981837    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.981912 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.981895    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-telemeter-client-tls\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.982054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.981930    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa947095-7d5c-4e06-9e46-d297fe162fd3-serving-certs-ca-bundle\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.982054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.981953    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-federate-client-tls\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.982054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.981977    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa947095-7d5c-4e06-9e46-d297fe162fd3-metrics-client-ca\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.982054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.982010    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klv4r\" (UniqueName: \"kubernetes.io/projected/aa947095-7d5c-4e06-9e46-d297fe162fd3-kube-api-access-klv4r\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.982609 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.982588    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa947095-7d5c-4e06-9e46-d297fe162fd3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.983230 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.983183    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa947095-7d5c-4e06-9e46-d297fe162fd3-serving-certs-ca-bundle\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.984196 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.984153    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa947095-7d5c-4e06-9e46-d297fe162fd3-metrics-client-ca\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.985379 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.985352    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.985480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.985360    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-secret-telemeter-client\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.986086 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.986062    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-telemeter-client-tls\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.986240 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.986217    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aa947095-7d5c-4e06-9e46-d297fe162fd3-federate-client-tls\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:42.992818 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:42.992796    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klv4r\" (UniqueName: \"kubernetes.io/projected/aa947095-7d5c-4e06-9e46-d297fe162fd3-kube-api-access-klv4r\") pod \"telemeter-client-fd6bd5bbd-j58w2\" (UID: \"aa947095-7d5c-4e06-9e46-d297fe162fd3\") " pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:43.019073 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:43.019036    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"]
Oct 30 20:37:43.207908 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:43.207664    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"
Oct 30 20:37:43.509363 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:43.509332    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2"]
Oct 30 20:37:43.546377 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:43.546348    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:37:43.669799 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:43.669759    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa947095_7d5c_4e06_9e46_d297fe162fd3.slice/crio-254798e255425edd6d90efebed83b73cb69bcdbdaee4bd417677e8d4b03d725b WatchSource:0}: Error finding container 254798e255425edd6d90efebed83b73cb69bcdbdaee4bd417677e8d4b03d725b: Status 404 returned error can't find the container with id 254798e255425edd6d90efebed83b73cb69bcdbdaee4bd417677e8d4b03d725b
Oct 30 20:37:43.985161 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:43.985079    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5" event={"ID":"07c3a947-1296-4e31-9d93-4fdb854fc889","Type":"ContainerStarted","Data":"33581b0caabcb988b02df2fb4ad0862a2f6cc2e871bc492a65c63a0af9486f5a"}
Oct 30 20:37:43.989288 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:43.989251    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c4b849c-pg2d4" event={"ID":"ef43ee01-2c9f-48e4-a1ac-9f21dc308579","Type":"ContainerStarted","Data":"c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09"}
Oct 30 20:37:43.993909 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:43.992411    4810 generic.go:356] "Generic (PLEG): container finished" podID="8a82d659-a274-4c7b-9935-5004f715b456" containerID="0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0" exitCode=0
Oct 30 20:37:43.994097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:43.994029    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerDied","Data":"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"}
Oct 30 20:37:44.012526 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.012165    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth" event={"ID":"af70ce71-5273-4ef0-908e-c374184d46a2","Type":"ContainerStarted","Data":"0482baa67ebf3e9c5f62cd906fe4ce286e01345bb5c5856dd01aa8a1fbcd6746"}
Oct 30 20:37:44.017734 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.017640    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2" event={"ID":"aa947095-7d5c-4e06-9e46-d297fe162fd3","Type":"ContainerStarted","Data":"254798e255425edd6d90efebed83b73cb69bcdbdaee4bd417677e8d4b03d725b"}
Oct 30 20:37:44.048883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.046388    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-857c4b849c-pg2d4" podStartSLOduration=1.5250270860000001 podStartE2EDuration="7.046365718s" totalImagesPullingTime="5.521338632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:44.015906933 +0000 UTC m=+30.507723475" watchObservedRunningTime="2026-10-30 20:37:44.046365718 +0000 UTC m=+30.538182263"
Oct 30 20:37:44.048883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.047297    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:37:44.069157 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.069114    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.070386 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.070349    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.076525    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.076709    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.076767    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.076851    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.076983    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.077073    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.077150    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.077185    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.077505    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.077581    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3cp611nc7ognl\""
Oct 30 20:37:44.077746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.077714    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 20:37:44.078976 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.078597    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wldr4\""
Oct 30 20:37:44.078976 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.078729    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 20:37:44.078976 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.078776    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 20:37:44.081495 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.081336    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 20:37:44.081495 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.081468    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200096    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200131    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-web-config\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200153    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200178    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200194    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200210    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200224    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200242    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200266    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200288    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200311    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200330    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200345    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200363    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200382    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200400    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200444    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config-out\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200472    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200488    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.203251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.200509    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swr5m\" (UniqueName: \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-kube-api-access-swr5m\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302109    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302179    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config-out\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302210    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302232    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302257    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swr5m\" (UniqueName: \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-kube-api-access-swr5m\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302307    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302336    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-web-config\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302397    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302465    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302498    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302522    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302543    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302566    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302606    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302637    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302658    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302686    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302705    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.302969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.302984    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.304054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.303326    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.304054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.303691    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.304480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.304450    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.304876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.304835    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.308455 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.305306    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.308455 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.307203    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.308455 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.307607    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.308455 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.307721    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.308455 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.307771    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.308455 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.307924    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.308455 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.308408    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.309108 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.309061    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config-out\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.312488 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.312092    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.312488 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.312267    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.312488 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.312280    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.312488 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.312449    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.312708 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.312683    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.312836 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.312810    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-web-config\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.314665 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.314634    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.316412 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.316387    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swr5m\" (UniqueName: \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-kube-api-access-swr5m\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.318069 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.318025    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.598492 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.598404    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:44.722351 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.721422    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lfl79"
Oct 30 20:37:44.920981 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:44.920878    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:37:44.933113 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:44.933071    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3e1b20_0cd4_4161_b4e0_0125f944439a.slice/crio-819473c68daee4b359c0edd56c6fac75eb6915086b7c2d063398399ee070502e WatchSource:0}: Error finding container 819473c68daee4b359c0edd56c6fac75eb6915086b7c2d063398399ee070502e: Status 404 returned error can't find the container with id 819473c68daee4b359c0edd56c6fac75eb6915086b7c2d063398399ee070502e
Oct 30 20:37:45.032904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:45.032753    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth" event={"ID":"af70ce71-5273-4ef0-908e-c374184d46a2","Type":"ContainerStarted","Data":"734df43aadcf307c5c13a664cf6726dfe5e20afd88cd57c946ca5f0ff155d547"}
Oct 30 20:37:45.032904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:45.032840    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth" event={"ID":"af70ce71-5273-4ef0-908e-c374184d46a2","Type":"ContainerStarted","Data":"d02f973db6296baac2111b8a23658872149711b3d7cf0bc81b7de68667d8d405"}
Oct 30 20:37:45.039977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:45.039899    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerStarted","Data":"819473c68daee4b359c0edd56c6fac75eb6915086b7c2d063398399ee070502e"}
Oct 30 20:37:46.048621 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.048540    4810 generic.go:356] "Generic (PLEG): container finished" podID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerID="3dcf20713c2b45729e55284bd77f1484a021952d8dab8056c8ad557aed12a42a" exitCode=0
Oct 30 20:37:46.049062 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.048616    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerDied","Data":"3dcf20713c2b45729e55284bd77f1484a021952d8dab8056c8ad557aed12a42a"}
Oct 30 20:37:46.504594 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.504101    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bdfd99cc5-zbvk2"]
Oct 30 20:37:46.533087 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.533053    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdfd99cc5-zbvk2"]
Oct 30 20:37:46.533239 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.533173    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.540187 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.540160    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 20:37:46.640394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.640354    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-oauth-serving-cert\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.640560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.640422    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c78cb\" (UniqueName: \"kubernetes.io/projected/3c202457-a065-42bd-b310-813616b4d8ef-kube-api-access-c78cb\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.640560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.640454    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-serving-cert\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.640560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.640488    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-console-config\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.640560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.640513    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c202457-a065-42bd-b310-813616b4d8ef-tmp\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.640778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.640577    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-oauth-config\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.640778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.640642    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-trusted-ca-bundle\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.640778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.640705    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-service-ca\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.741982 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.741953    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-trusted-ca-bundle\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.742125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.742006    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-service-ca\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.742125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.742079    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-oauth-serving-cert\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.742232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.742126    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c78cb\" (UniqueName: \"kubernetes.io/projected/3c202457-a065-42bd-b310-813616b4d8ef-kube-api-access-c78cb\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.742232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.742161    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-serving-cert\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.742232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.742197    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-console-config\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.742232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.742220    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c202457-a065-42bd-b310-813616b4d8ef-tmp\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.742436 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.742287    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-oauth-config\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.742899 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.742848    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-service-ca\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.743007 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.742906    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-oauth-serving-cert\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.743007 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.742988    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-trusted-ca-bundle\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.743117 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.743037    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c202457-a065-42bd-b310-813616b4d8ef-tmp\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.743177 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.743113    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-console-config\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.745161 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.745134    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-serving-cert\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.745411 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.745379    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-oauth-config\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.751469 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.751428    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c78cb\" (UniqueName: \"kubernetes.io/projected/3c202457-a065-42bd-b310-813616b4d8ef-kube-api-access-c78cb\") pod \"console-6bdfd99cc5-zbvk2\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") " pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:46.868866 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:46.868835    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:47.056468 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.056383    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td" event={"ID":"05f34d74-7115-42fb-a288-27bf8ef7c3e8","Type":"ContainerStarted","Data":"791d396f2d245840ba9c7e8c6cf2d55743dac4eb2d224852d31178a54e4af3b8"}
Oct 30 20:37:47.059927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.059895    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth" event={"ID":"af70ce71-5273-4ef0-908e-c374184d46a2","Type":"ContainerStarted","Data":"724ebe6f89b1a63c3160a48ba34b3d92d4c61d763075652fca9de5943a26290d"}
Oct 30 20:37:47.060072 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.059970    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth" event={"ID":"af70ce71-5273-4ef0-908e-c374184d46a2","Type":"ContainerStarted","Data":"d6513aab0b2b29a52a2e91c4bbc33352122df1850d339e3acc8a21b72ac44f37"}
Oct 30 20:37:47.061738 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.061715    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5" event={"ID":"07c3a947-1296-4e31-9d93-4fdb854fc889","Type":"ContainerStarted","Data":"fc2c3db492f1b02781e13c81811ded8f78b3eed9c33fd2b2c3c4d96642792ec0"}
Oct 30 20:37:47.062044 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.061995    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"
Oct 30 20:37:47.067199 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.067176    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5"
Oct 30 20:37:47.079924 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.079848    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td" podStartSLOduration=1.085717701 podStartE2EDuration="5.079836394s" totalImagesPullingTime="3.994118693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:47.079256216 +0000 UTC m=+33.571072762" watchObservedRunningTime="2026-10-30 20:37:47.079836394 +0000 UTC m=+33.571652941"
Oct 30 20:37:47.098369 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.098325    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-84kx5" podStartSLOduration=1.446849413 podStartE2EDuration="5.098312991s" totalImagesPullingTime="3.651463578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:47.095992888 +0000 UTC m=+33.587809438" watchObservedRunningTime="2026-10-30 20:37:47.098312991 +0000 UTC m=+33.590129536"
Oct 30 20:37:47.463817 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.463598    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdfd99cc5-zbvk2"]
Oct 30 20:37:47.468324 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:47.468196    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c202457_a065_42bd_b310_813616b4d8ef.slice/crio-68496bbf6bb7b1671177f30be381bfad0cd13e63ea57ddbb1c5c561bf4f21aba WatchSource:0}: Error finding container 68496bbf6bb7b1671177f30be381bfad0cd13e63ea57ddbb1c5c561bf4f21aba: Status 404 returned error can't find the container with id 68496bbf6bb7b1671177f30be381bfad0cd13e63ea57ddbb1c5c561bf4f21aba
Oct 30 20:37:47.878999 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.878972    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:47.880578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.879229    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:47.884787 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:47.884765    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:48.072703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.072600    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerStarted","Data":"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"}
Oct 30 20:37:48.073283 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.072709    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerStarted","Data":"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"}
Oct 30 20:37:48.073283 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.072756    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerStarted","Data":"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"}
Oct 30 20:37:48.073283 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.072796    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerStarted","Data":"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"}
Oct 30 20:37:48.073283 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.072840    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerStarted","Data":"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"}
Oct 30 20:37:48.073283 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.072909    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerStarted","Data":"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"}
Oct 30 20:37:48.076214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.076189    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth" event={"ID":"af70ce71-5273-4ef0-908e-c374184d46a2","Type":"ContainerStarted","Data":"8be7f6892bdbe7a181adfc3ebd4a3d911e9f18fc1fa66442c45940d726d19283"}
Oct 30 20:37:48.076341 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.076326    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:48.080662 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.080634    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2" event={"ID":"aa947095-7d5c-4e06-9e46-d297fe162fd3","Type":"ContainerStarted","Data":"c6c27d02d817ce14293afbb7fd3804ea1e6f8006599e97936ee23e9941065b0b"}
Oct 30 20:37:48.080777 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.080693    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2" event={"ID":"aa947095-7d5c-4e06-9e46-d297fe162fd3","Type":"ContainerStarted","Data":"1b6994bc067e50c1b913b32e09385dede0e569e253b4bb38ecbfd3f663c800f2"}
Oct 30 20:37:48.080777 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.080723    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2" event={"ID":"aa947095-7d5c-4e06-9e46-d297fe162fd3","Type":"ContainerStarted","Data":"f4411ade02095f99e43414581b7e9760132f79e43162dfaec0277216538f95d9"}
Oct 30 20:37:48.083615 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.083591    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdfd99cc5-zbvk2" event={"ID":"3c202457-a065-42bd-b310-813616b4d8ef","Type":"ContainerStarted","Data":"f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110"}
Oct 30 20:37:48.083713 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.083631    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdfd99cc5-zbvk2" event={"ID":"3c202457-a065-42bd-b310-813616b4d8ef","Type":"ContainerStarted","Data":"68496bbf6bb7b1671177f30be381bfad0cd13e63ea57ddbb1c5c561bf4f21aba"}
Oct 30 20:37:48.089057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.089037    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:37:48.107936 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.107887    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.79465874 podStartE2EDuration="10.107875331s" totalImagesPullingTime="5.300632041s" totalInitContainerRuntime="12.58455ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:48.10677345 +0000 UTC m=+34.598590022" watchObservedRunningTime="2026-10-30 20:37:48.107875331 +0000 UTC m=+34.599691867"
Oct 30 20:37:48.133739 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.132695    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6bdfd99cc5-zbvk2" podStartSLOduration=2.132682258 podStartE2EDuration="2.132682258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:48.132254858 +0000 UTC m=+34.624071404" watchObservedRunningTime="2026-10-30 20:37:48.132682258 +0000 UTC m=+34.624498802"
Oct 30 20:37:48.217781 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.217211    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-fd6bd5bbd-j58w2" podStartSLOduration=2.760675476 podStartE2EDuration="6.217192026s" totalImagesPullingTime="3.45651655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:48.216971781 +0000 UTC m=+34.708788327" watchObservedRunningTime="2026-10-30 20:37:48.217192026 +0000 UTC m=+34.709008572"
Oct 30 20:37:48.220619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:48.219750    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth" podStartSLOduration=3.8977524900000002 podStartE2EDuration="9.219727488s" totalImagesPullingTime="5.321974998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:48.191884852 +0000 UTC m=+34.683701397" watchObservedRunningTime="2026-10-30 20:37:48.219727488 +0000 UTC m=+34.711544034"
Oct 30 20:37:49.099925 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:49.099890    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-744b9bc9d7-grkth"
Oct 30 20:37:51.854231 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:51.854206    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56dfd5c8cd-9dzxh"
Oct 30 20:37:51.967148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:51.967125    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-948b95b59-tdzdl"]
Oct 30 20:37:52.138408 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:52.138374    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerStarted","Data":"b8e493632cf15885137f7774c7c9f5b0e3e90c1b49d73c14c665b3799110a0be"}
Oct 30 20:37:52.138566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:52.138455    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerStarted","Data":"9dbd5fceec594eebfa82fcb2449d989dd01fecc40416fbc2a56a431a8867a8bb"}
Oct 30 20:37:52.138566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:52.138484    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerStarted","Data":"0de84fa73f287378c1e61b1c1b279907eed8c5e6fb0e64eb619ee08f14b8fe22"}
Oct 30 20:37:52.138566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:52.138506    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerStarted","Data":"5e1941c360b1814b26d21f483399c75b9270f1040ad22b52a3378f7672df82c8"}
Oct 30 20:37:52.138566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:52.138529    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerStarted","Data":"700d5b7cbee1ac9997c4fcc5d0bb6bfa55a296ad47610c742fec83fac6a6def8"}
Oct 30 20:37:53.166544 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.160735    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerStarted","Data":"52afeedf8a175f32b712f288ff2dee37aca5b7ac5f6aa6b980516569de6e722b"}
Oct 30 20:37:53.212174 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.211665    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.849842884 podStartE2EDuration="9.211650667s" totalImagesPullingTime="5.332019738s" totalInitContainerRuntime="29.788045ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:53.209127573 +0000 UTC m=+39.700944130" watchObservedRunningTime="2026-10-30 20:37:53.211650667 +0000 UTC m=+39.703467233"
Oct 30 20:37:53.715399 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.715360    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bdfd99cc5-zbvk2"]
Oct 30 20:37:53.742721 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.742689    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d9698fc4f-8fzsh"]
Oct 30 20:37:53.803947 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.789002    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9698fc4f-8fzsh"]
Oct 30 20:37:53.803947 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.789150    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:53.901216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.900707    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqm54\" (UniqueName: \"kubernetes.io/projected/ae527ee9-5da3-4627-8176-5226765ad684-kube-api-access-qqm54\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:53.901216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.900788    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-oauth-serving-cert\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:53.901216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.900831    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-service-ca\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:53.901216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.900894    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-console-config\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:53.901216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.900935    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-oauth-config\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:53.901216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.900994    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae527ee9-5da3-4627-8176-5226765ad684-tmp\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:53.901216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.901071    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-serving-cert\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:53.901216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:53.901110    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-trusted-ca-bundle\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.001993 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.001909    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-trusted-ca-bundle\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.002164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.002006    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqm54\" (UniqueName: \"kubernetes.io/projected/ae527ee9-5da3-4627-8176-5226765ad684-kube-api-access-qqm54\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.002164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.002068    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-oauth-serving-cert\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.002164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.002107    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-service-ca\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.002164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.002151    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-console-config\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.002403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.002192    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-oauth-config\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.002403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.002253    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae527ee9-5da3-4627-8176-5226765ad684-tmp\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.002403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.002334    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-serving-cert\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.003050 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.002975    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-service-ca\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.003439 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.003373    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-console-config\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.003439 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.002983    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-trusted-ca-bundle\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.003439 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.003381    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae527ee9-5da3-4627-8176-5226765ad684-tmp\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.003663 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.003588    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-oauth-serving-cert\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.005524 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.005497    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-serving-cert\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.007273 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.007252    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-oauth-config\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.013259 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.013234    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqm54\" (UniqueName: \"kubernetes.io/projected/ae527ee9-5da3-4627-8176-5226765ad684-kube-api-access-qqm54\") pod \"console-5d9698fc4f-8fzsh\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") " pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.150299 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.150062    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:37:54.184813 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.182716    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-g57td" event={"ID":"d9f77f82-84fd-496c-994e-7fdc2a018af3","Type":"ContainerStarted","Data":"dba0fd9ebc0e9553500eedcf0fd1017da6d8faa708424265790fd1eb8b592820"}
Oct 30 20:37:54.184813 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.183132    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-g57td"
Oct 30 20:37:54.194010 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.193981    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-g57td"
Oct 30 20:37:54.200492 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.200435    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-g57td" podStartSLOduration=2.593811022 podStartE2EDuration="25.200418542s" totalImagesPullingTime="22.60660752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:54.199822822 +0000 UTC m=+40.691639369" watchObservedRunningTime="2026-10-30 20:37:54.200418542 +0000 UTC m=+40.692235087"
Oct 30 20:37:54.515251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.515219    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9698fc4f-8fzsh"]
Oct 30 20:37:54.521302 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:37:54.521271    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae527ee9_5da3_4627_8176_5226765ad684.slice/crio-de0651acf3d4acbeddd6a247eadec240562ea326844154f484dbb14a2e603d25 WatchSource:0}: Error finding container de0651acf3d4acbeddd6a247eadec240562ea326844154f484dbb14a2e603d25: Status 404 returned error can't find the container with id de0651acf3d4acbeddd6a247eadec240562ea326844154f484dbb14a2e603d25
Oct 30 20:37:54.599373 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.599269    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:37:54.923003 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:54.922117    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz" podUID="204ba748-a514-4772-9fba-dfafd3eddcf7" containerName="registry" containerID="cri-o://fdbeaea20112a0a8345b6226da3a79e3987b2360a641c1eb15ef28c628c7a6b0" gracePeriod=30
Oct 30 20:37:55.202795 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:37:55.202725    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:37:55.207763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.207719    4810 generic.go:356] "Generic (PLEG): container finished" podID="204ba748-a514-4772-9fba-dfafd3eddcf7" containerID="fdbeaea20112a0a8345b6226da3a79e3987b2360a641c1eb15ef28c628c7a6b0" exitCode=0
Oct 30 20:37:55.207930 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.207793    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz" event={"ID":"204ba748-a514-4772-9fba-dfafd3eddcf7","Type":"ContainerDied","Data":"fdbeaea20112a0a8345b6226da3a79e3987b2360a641c1eb15ef28c628c7a6b0"}
Oct 30 20:37:55.211775 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.211735    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9698fc4f-8fzsh" event={"ID":"ae527ee9-5da3-4627-8176-5226765ad684","Type":"ContainerStarted","Data":"c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae"}
Oct 30 20:37:55.211932 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.211787    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9698fc4f-8fzsh" event={"ID":"ae527ee9-5da3-4627-8176-5226765ad684","Type":"ContainerStarted","Data":"de0651acf3d4acbeddd6a247eadec240562ea326844154f484dbb14a2e603d25"}
Oct 30 20:37:55.234344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.232792    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d9698fc4f-8fzsh" podStartSLOduration=2.232772935 podStartE2EDuration="2.232772935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:55.231353515 +0000 UTC m=+41.723170054" watchObservedRunningTime="2026-10-30 20:37:55.232772935 +0000 UTC m=+41.724589482"
Oct 30 20:37:55.353195 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.353125    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:55.432974 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.432935    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-kube-api-access-v7b7j\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-kube-api-access-v7b7j\") pod \"204ba748-a514-4772-9fba-dfafd3eddcf7\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") "
Oct 30 20:37:55.433258 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.433235    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") pod \"204ba748-a514-4772-9fba-dfafd3eddcf7\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") "
Oct 30 20:37:55.433417 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.433394    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-image-registry-private-configuration\") pod \"204ba748-a514-4772-9fba-dfafd3eddcf7\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") "
Oct 30 20:37:55.433554 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.433539    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-trusted-ca\") pod \"204ba748-a514-4772-9fba-dfafd3eddcf7\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") "
Oct 30 20:37:55.433689 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.433673    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-installation-pull-secrets\") pod \"204ba748-a514-4772-9fba-dfafd3eddcf7\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") "
Oct 30 20:37:55.433886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.433869    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-certificates\") pod \"204ba748-a514-4772-9fba-dfafd3eddcf7\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") "
Oct 30 20:37:55.434048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.434031    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204ba748-a514-4772-9fba-dfafd3eddcf7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/204ba748-a514-4772-9fba-dfafd3eddcf7-ca-trust-extracted\") pod \"204ba748-a514-4772-9fba-dfafd3eddcf7\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") "
Oct 30 20:37:55.434208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.434192    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-bound-sa-token\") pod \"204ba748-a514-4772-9fba-dfafd3eddcf7\" (UID: \"204ba748-a514-4772-9fba-dfafd3eddcf7\") "
Oct 30 20:37:55.434995 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.434957    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-certificates" pod "204ba748-a514-4772-9fba-dfafd3eddcf7" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:37:55.435200 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.435171    4810 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-certificates\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:55.438006 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.437980    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-bound-sa-token" pod "204ba748-a514-4772-9fba-dfafd3eddcf7" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:37:55.440993 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.440963    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls" pod "204ba748-a514-4772-9fba-dfafd3eddcf7" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:37:55.445384 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.444268    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204ba748-a514-4772-9fba-dfafd3eddcf7-ca-trust-extracted" pod "204ba748-a514-4772-9fba-dfafd3eddcf7" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:37:55.445533 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.445426    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-image-registry-private-configuration" pod "204ba748-a514-4772-9fba-dfafd3eddcf7" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:37:55.446900 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.446871    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-trusted-ca" pod "204ba748-a514-4772-9fba-dfafd3eddcf7" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:37:55.447332 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.447311    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-kube-api-access-v7b7j" pod "204ba748-a514-4772-9fba-dfafd3eddcf7" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7"). InnerVolumeSpecName "kube-api-access-v7b7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:37:55.450583 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.450556    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-installation-pull-secrets" pod "204ba748-a514-4772-9fba-dfafd3eddcf7" (UID: "204ba748-a514-4772-9fba-dfafd3eddcf7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:37:55.540561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.537927    4810 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/204ba748-a514-4772-9fba-dfafd3eddcf7-ca-trust-extracted\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:55.540561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.537965    4810 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-bound-sa-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:55.540561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.537982    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7b7j\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-kube-api-access-v7b7j\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:55.540561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.537999    4810 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/204ba748-a514-4772-9fba-dfafd3eddcf7-registry-tls\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:55.540561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.538016    4810 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-image-registry-private-configuration\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:55.540561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.538033    4810 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/204ba748-a514-4772-9fba-dfafd3eddcf7-trusted-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:55.540561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:55.538048    4810 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/204ba748-a514-4772-9fba-dfafd3eddcf7-installation-pull-secrets\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:56.224516 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:56.224453    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz"
Oct 30 20:37:56.224516 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:56.224502    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfdcc7f98-zlptz" event={"ID":"204ba748-a514-4772-9fba-dfafd3eddcf7","Type":"ContainerDied","Data":"f4e223ec55c14e792883a477e3416314f4b70dff96cb11cf7d0d84c0402b0bd9"}
Oct 30 20:37:56.225161 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:56.224572    4810 scope.go:122] "RemoveContainer" containerID="fdbeaea20112a0a8345b6226da3a79e3987b2360a641c1eb15ef28c628c7a6b0"
Oct 30 20:37:56.250650 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:56.250592    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bfdcc7f98-zlptz"]
Oct 30 20:37:56.259583 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:56.258800    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bfdcc7f98-zlptz"]
Oct 30 20:37:56.869902 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:56.869869    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:37:57.663100 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:37:57.663070    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204ba748-a514-4772-9fba-dfafd3eddcf7" path="/var/lib/kubelet/pods/204ba748-a514-4772-9fba-dfafd3eddcf7/volumes"
Oct 30 20:38:02.243084 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.243049    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7k6rz-jhhmx"]
Oct 30 20:38:02.243606 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.243248    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-7k6rz-jhhmx" podUID="2d2edfa6-676f-47ab-af10-37be9080c821" containerName="insights-gathering" containerID="cri-o://1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef" gracePeriod=30
Oct 30 20:38:02.439042 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.439011    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:38:02.439219 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.439070    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:38:02.574099 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.574074    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:38:02.626677 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.626645    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-service-ca-bundle\") pod \"2d2edfa6-676f-47ab-af10-37be9080c821\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") "
Oct 30 20:38:02.626888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.626752    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2edfa6-676f-47ab-af10-37be9080c821-kube-api-access-xqcxr\" (UniqueName: \"kubernetes.io/projected/2d2edfa6-676f-47ab-af10-37be9080c821-kube-api-access-xqcxr\") pod \"2d2edfa6-676f-47ab-af10-37be9080c821\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") "
Oct 30 20:38:02.626888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.626825    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2edfa6-676f-47ab-af10-37be9080c821-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2edfa6-676f-47ab-af10-37be9080c821-archives-path\") pod \"2d2edfa6-676f-47ab-af10-37be9080c821\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") "
Oct 30 20:38:02.627020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.626891    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-trusted-ca-bundle\") pod \"2d2edfa6-676f-47ab-af10-37be9080c821\" (UID: \"2d2edfa6-676f-47ab-af10-37be9080c821\") "
Oct 30 20:38:02.627093 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.627064    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-service-ca-bundle" pod "2d2edfa6-676f-47ab-af10-37be9080c821" (UID: "2d2edfa6-676f-47ab-af10-37be9080c821"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:02.627190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.627165    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2edfa6-676f-47ab-af10-37be9080c821-archives-path" pod "2d2edfa6-676f-47ab-af10-37be9080c821" (UID: "2d2edfa6-676f-47ab-af10-37be9080c821"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:02.627268 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.627245    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-trusted-ca-bundle" pod "2d2edfa6-676f-47ab-af10-37be9080c821" (UID: "2d2edfa6-676f-47ab-af10-37be9080c821"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:02.627324 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.627277    4810 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-service-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:02.627324 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.627299    4810 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2d2edfa6-676f-47ab-af10-37be9080c821-archives-path\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:02.629115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.629081    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2edfa6-676f-47ab-af10-37be9080c821-kube-api-access-xqcxr" pod "2d2edfa6-676f-47ab-af10-37be9080c821" (UID: "2d2edfa6-676f-47ab-af10-37be9080c821"). InnerVolumeSpecName "kube-api-access-xqcxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:02.728205 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.728176    4810 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d2edfa6-676f-47ab-af10-37be9080c821-trusted-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:02.728205 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:02.728206    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqcxr\" (UniqueName: \"kubernetes.io/projected/2d2edfa6-676f-47ab-af10-37be9080c821-kube-api-access-xqcxr\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:03.268219 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:03.268186    4810 generic.go:356] "Generic (PLEG): container finished" podID="2d2edfa6-676f-47ab-af10-37be9080c821" containerID="1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef" exitCode=2
Oct 30 20:38:03.268729 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:03.268231    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7k6rz-jhhmx" event={"ID":"2d2edfa6-676f-47ab-af10-37be9080c821","Type":"ContainerDied","Data":"1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef"}
Oct 30 20:38:03.268729 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:03.268248    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7k6rz-jhhmx"
Oct 30 20:38:03.268729 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:03.268271    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7k6rz-jhhmx" event={"ID":"2d2edfa6-676f-47ab-af10-37be9080c821","Type":"ContainerDied","Data":"5b7e77e725cde65cecf7918764087c2c53584f845186c677033925e14d6db3b7"}
Oct 30 20:38:03.268729 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:03.268289    4810 scope.go:122] "RemoveContainer" containerID="1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef"
Oct 30 20:38:03.278327 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:03.278312    4810 scope.go:122] "RemoveContainer" containerID="1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef"
Oct 30 20:38:03.278595 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:03.278576    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef\": container with ID starting with 1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef not found: ID does not exist" containerID="1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef"
Oct 30 20:38:03.278667 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:03.278607    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef"} err="failed to get container status \"1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef\": rpc error: code = NotFound desc = could not find container \"1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef\": container with ID starting with 1f6b73fe942b2fd062696417f6843f916e2d02baad3ab876a6f2db5e8048e8ef not found: ID does not exist"
Oct 30 20:38:03.294686 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:03.294661    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7k6rz-jhhmx"]
Oct 30 20:38:03.299357 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:03.299336    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-7k6rz-jhhmx"]
Oct 30 20:38:03.662703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:03.662665    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2edfa6-676f-47ab-af10-37be9080c821" path="/var/lib/kubelet/pods/2d2edfa6-676f-47ab-af10-37be9080c821/volumes"
Oct 30 20:38:04.151819 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:04.151787    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:38:04.152002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:04.151851    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:38:04.156663 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:04.156643    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:38:04.278788 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:04.278762    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:38:04.349295 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:04.349257    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857c4b849c-pg2d4"]
Oct 30 20:38:16.997090 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:16.997029    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-948b95b59-tdzdl" podUID="4dd510fb-f40e-401c-845f-23b9d750dc68" containerName="registry" containerID="cri-o://668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0" gracePeriod=30
Oct 30 20:38:17.311502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.311475    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:38:17.337674 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.337646    4810 generic.go:356] "Generic (PLEG): container finished" podID="4dd510fb-f40e-401c-845f-23b9d750dc68" containerID="668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0" exitCode=0
Oct 30 20:38:17.337799 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.337683    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-948b95b59-tdzdl"
Oct 30 20:38:17.337799 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.337731    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-948b95b59-tdzdl" event={"ID":"4dd510fb-f40e-401c-845f-23b9d750dc68","Type":"ContainerDied","Data":"668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0"}
Oct 30 20:38:17.337799 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.337782    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-948b95b59-tdzdl" event={"ID":"4dd510fb-f40e-401c-845f-23b9d750dc68","Type":"ContainerDied","Data":"c75a50aeb54ce34fc6bf3a3689711c09825c1ff790391b0e2d031c0e94530423"}
Oct 30 20:38:17.337923 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.337811    4810 scope.go:122] "RemoveContainer" containerID="668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0"
Oct 30 20:38:17.351933 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.351915    4810 scope.go:122] "RemoveContainer" containerID="668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0"
Oct 30 20:38:17.352274 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:17.352257    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0\": container with ID starting with 668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0 not found: ID does not exist" containerID="668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0"
Oct 30 20:38:17.352344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.352288    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0"} err="failed to get container status \"668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0\": rpc error: code = NotFound desc = could not find container \"668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0\": container with ID starting with 668c9f3f4ba5913c5cf137b477f9fca9fe1e64a2409b0875e932a91cc5b17bf0 not found: ID does not exist"
Oct 30 20:38:17.466534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.466507    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-installation-pull-secrets\") pod \"4dd510fb-f40e-401c-845f-23b9d750dc68\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") "
Oct 30 20:38:17.466727 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.466574    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-trusted-ca\") pod \"4dd510fb-f40e-401c-845f-23b9d750dc68\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") "
Oct 30 20:38:17.466797 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.466751    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-certificates\") pod \"4dd510fb-f40e-401c-845f-23b9d750dc68\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") "
Oct 30 20:38:17.466883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.466812    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-image-registry-private-configuration\") pod \"4dd510fb-f40e-401c-845f-23b9d750dc68\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") "
Oct 30 20:38:17.466948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.466881    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") pod \"4dd510fb-f40e-401c-845f-23b9d750dc68\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") "
Oct 30 20:38:17.466948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.466934    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-kube-api-access-jvtbb\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-kube-api-access-jvtbb\") pod \"4dd510fb-f40e-401c-845f-23b9d750dc68\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") "
Oct 30 20:38:17.467155 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.467129    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-trusted-ca" pod "4dd510fb-f40e-401c-845f-23b9d750dc68" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:17.467249 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.467165    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-certificates" pod "4dd510fb-f40e-401c-845f-23b9d750dc68" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:17.467462 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.467439    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-bound-sa-token\") pod \"4dd510fb-f40e-401c-845f-23b9d750dc68\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") "
Oct 30 20:38:17.467658 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.467638    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd510fb-f40e-401c-845f-23b9d750dc68-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dd510fb-f40e-401c-845f-23b9d750dc68-ca-trust-extracted\") pod \"4dd510fb-f40e-401c-845f-23b9d750dc68\" (UID: \"4dd510fb-f40e-401c-845f-23b9d750dc68\") "
Oct 30 20:38:17.468281 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.468260    4810 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-trusted-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:17.468421 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.468404    4810 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-certificates\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:17.469365 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.469292    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-installation-pull-secrets" pod "4dd510fb-f40e-401c-845f-23b9d750dc68" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:17.469575 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.469551    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls" pod "4dd510fb-f40e-401c-845f-23b9d750dc68" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:17.469852 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.469829    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-kube-api-access-jvtbb" pod "4dd510fb-f40e-401c-845f-23b9d750dc68" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68"). InnerVolumeSpecName "kube-api-access-jvtbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:17.469933 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.469888    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-bound-sa-token" pod "4dd510fb-f40e-401c-845f-23b9d750dc68" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:17.470052 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.470036    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-image-registry-private-configuration" pod "4dd510fb-f40e-401c-845f-23b9d750dc68" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:17.477016 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.476991    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd510fb-f40e-401c-845f-23b9d750dc68-ca-trust-extracted" pod "4dd510fb-f40e-401c-845f-23b9d750dc68" (UID: "4dd510fb-f40e-401c-845f-23b9d750dc68"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:17.569775 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.569688    4810 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-bound-sa-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:17.569775 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.569720    4810 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dd510fb-f40e-401c-845f-23b9d750dc68-ca-trust-extracted\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:17.569775 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.569732    4810 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-installation-pull-secrets\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:17.569775 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.569744    4810 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dd510fb-f40e-401c-845f-23b9d750dc68-image-registry-private-configuration\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:17.569775 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.569754    4810 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-registry-tls\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:17.569775 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.569763    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvtbb\" (UniqueName: \"kubernetes.io/projected/4dd510fb-f40e-401c-845f-23b9d750dc68-kube-api-access-jvtbb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:17.668541 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.668512    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-948b95b59-tdzdl"]
Oct 30 20:38:17.672906 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:17.672883    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-948b95b59-tdzdl"]
Oct 30 20:38:18.740222 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:18.740182    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6bdfd99cc5-zbvk2" podUID="3c202457-a065-42bd-b310-813616b4d8ef" containerName="console" containerID="cri-o://f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110" gracePeriod=15
Oct 30 20:38:19.067986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.067956    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:38:19.193137 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193101    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c202457-a065-42bd-b310-813616b4d8ef-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c202457-a065-42bd-b310-813616b4d8ef-tmp\") pod \"3c202457-a065-42bd-b310-813616b4d8ef\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") "
Oct 30 20:38:19.193317 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193151    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-oauth-serving-cert\") pod \"3c202457-a065-42bd-b310-813616b4d8ef\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") "
Oct 30 20:38:19.193317 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193228    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-console-config\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-console-config\") pod \"3c202457-a065-42bd-b310-813616b4d8ef\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") "
Oct 30 20:38:19.193317 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193282    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-serving-cert\") pod \"3c202457-a065-42bd-b310-813616b4d8ef\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") "
Oct 30 20:38:19.193484 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193340    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-oauth-config\") pod \"3c202457-a065-42bd-b310-813616b4d8ef\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") "
Oct 30 20:38:19.193484 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193379    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-service-ca\") pod \"3c202457-a065-42bd-b310-813616b4d8ef\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") "
Oct 30 20:38:19.193484 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193385    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c202457-a065-42bd-b310-813616b4d8ef-tmp" pod "3c202457-a065-42bd-b310-813616b4d8ef" (UID: "3c202457-a065-42bd-b310-813616b4d8ef"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:19.193484 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193451    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c202457-a065-42bd-b310-813616b4d8ef-kube-api-access-c78cb\" (UniqueName: \"kubernetes.io/projected/3c202457-a065-42bd-b310-813616b4d8ef-kube-api-access-c78cb\") pod \"3c202457-a065-42bd-b310-813616b4d8ef\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") "
Oct 30 20:38:19.193664 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193484    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-trusted-ca-bundle\") pod \"3c202457-a065-42bd-b310-813616b4d8ef\" (UID: \"3c202457-a065-42bd-b310-813616b4d8ef\") "
Oct 30 20:38:19.193664 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193575    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-oauth-serving-cert" pod "3c202457-a065-42bd-b310-813616b4d8ef" (UID: "3c202457-a065-42bd-b310-813616b4d8ef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:19.193773 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193748    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-console-config" pod "3c202457-a065-42bd-b310-813616b4d8ef" (UID: "3c202457-a065-42bd-b310-813616b4d8ef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:19.193841 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193769    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-service-ca" pod "3c202457-a065-42bd-b310-813616b4d8ef" (UID: "3c202457-a065-42bd-b310-813616b4d8ef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:19.193908 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193832    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-trusted-ca-bundle" pod "3c202457-a065-42bd-b310-813616b4d8ef" (UID: "3c202457-a065-42bd-b310-813616b4d8ef"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:19.193944 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193923    4810 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:19.193977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193943    4810 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c202457-a065-42bd-b310-813616b4d8ef-tmp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:19.193977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193959    4810 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-oauth-serving-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:19.193977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.193973    4810 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-console-config\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:19.195624 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.195594    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-oauth-config" pod "3c202457-a065-42bd-b310-813616b4d8ef" (UID: "3c202457-a065-42bd-b310-813616b4d8ef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:19.195727 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.195635    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c202457-a065-42bd-b310-813616b4d8ef-kube-api-access-c78cb" pod "3c202457-a065-42bd-b310-813616b4d8ef" (UID: "3c202457-a065-42bd-b310-813616b4d8ef"). InnerVolumeSpecName "kube-api-access-c78cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:19.195727 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.195638    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-serving-cert" pod "3c202457-a065-42bd-b310-813616b4d8ef" (UID: "3c202457-a065-42bd-b310-813616b4d8ef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:19.295170 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.295102    4810 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-serving-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:19.295170 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.295129    4810 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c202457-a065-42bd-b310-813616b4d8ef-console-oauth-config\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:19.295170 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.295140    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c78cb\" (UniqueName: \"kubernetes.io/projected/3c202457-a065-42bd-b310-813616b4d8ef-kube-api-access-c78cb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:19.295170 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.295150    4810 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c202457-a065-42bd-b310-813616b4d8ef-trusted-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:19.350156 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.350126    4810 generic.go:356] "Generic (PLEG): container finished" podID="3c202457-a065-42bd-b310-813616b4d8ef" containerID="f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110" exitCode=0
Oct 30 20:38:19.350277 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.350177    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdfd99cc5-zbvk2"
Oct 30 20:38:19.350277 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.350190    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdfd99cc5-zbvk2" event={"ID":"3c202457-a065-42bd-b310-813616b4d8ef","Type":"ContainerDied","Data":"f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110"}
Oct 30 20:38:19.350277 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.350251    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdfd99cc5-zbvk2" event={"ID":"3c202457-a065-42bd-b310-813616b4d8ef","Type":"ContainerDied","Data":"68496bbf6bb7b1671177f30be381bfad0cd13e63ea57ddbb1c5c561bf4f21aba"}
Oct 30 20:38:19.350277 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.350270    4810 scope.go:122] "RemoveContainer" containerID="f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110"
Oct 30 20:38:19.360025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.360009    4810 scope.go:122] "RemoveContainer" containerID="f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110"
Oct 30 20:38:19.360270 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:19.360253    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110\": container with ID starting with f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110 not found: ID does not exist" containerID="f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110"
Oct 30 20:38:19.360334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.360277    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110"} err="failed to get container status \"f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110\": rpc error: code = NotFound desc = could not find container \"f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110\": container with ID starting with f169e0f00e59f1419a31cbc1aa019a587f271fbd3e507d99245f2fa62f37c110 not found: ID does not exist"
Oct 30 20:38:19.373378 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.373352    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bdfd99cc5-zbvk2"]
Oct 30 20:38:19.378074 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.378056    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bdfd99cc5-zbvk2"]
Oct 30 20:38:19.660595 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.660567    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c202457-a065-42bd-b310-813616b4d8ef" path="/var/lib/kubelet/pods/3c202457-a065-42bd-b310-813616b4d8ef/volumes"
Oct 30 20:38:19.661012 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:19.660998    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd510fb-f40e-401c-845f-23b9d750dc68" path="/var/lib/kubelet/pods/4dd510fb-f40e-401c-845f-23b9d750dc68/volumes"
Oct 30 20:38:22.443794 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:22.443765    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:38:22.447402 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:22.447385    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f8657c7b9-5v8td"
Oct 30 20:38:25.155327 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:25.155290    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:38:29.374001 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.373906    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-857c4b849c-pg2d4" podUID="ef43ee01-2c9f-48e4-a1ac-9f21dc308579" containerName="console" containerID="cri-o://c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09" gracePeriod=15
Oct 30 20:38:29.406293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.406265    4810 generic.go:356] "Generic (PLEG): container finished" podID="b3e2fc05-6adb-4c94-a124-aca4483a4e13" containerID="79ac9a599cf3242c80b15d85d7c35581883bed2757c24df1d5a15483a7b2e57b" exitCode=0
Oct 30 20:38:29.406412 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.406340    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rbf5z" event={"ID":"b3e2fc05-6adb-4c94-a124-aca4483a4e13","Type":"ContainerDied","Data":"79ac9a599cf3242c80b15d85d7c35581883bed2757c24df1d5a15483a7b2e57b"}
Oct 30 20:38:29.406567 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.406438    4810 scope.go:122] "RemoveContainer" containerID="5920e6d7de20c10a6942708c81db189a50991832bcff606858220d8dcd1b7e33"
Oct 30 20:38:29.406825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.406809    4810 scope.go:122] "RemoveContainer" containerID="79ac9a599cf3242c80b15d85d7c35581883bed2757c24df1d5a15483a7b2e57b"
Oct 30 20:38:29.407080 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:29.407057    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-rbf5z_openshift-insights(b3e2fc05-6adb-4c94-a124-aca4483a4e13)\"" pod="openshift-insights/insights-operator-7c4986d449-rbf5z" podUID="b3e2fc05-6adb-4c94-a124-aca4483a4e13"
Oct 30 20:38:29.709941 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.709914    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:38:29.791500 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.791469    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-service-ca\") pod \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") "
Oct 30 20:38:29.791645 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.791517    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-tmp\") pod \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") "
Oct 30 20:38:29.791645 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.791576    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-oauth-config\") pod \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") "
Oct 30 20:38:29.791645 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.791609    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-config\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-config\") pod \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") "
Oct 30 20:38:29.791783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.791648    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-serving-cert\") pod \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") "
Oct 30 20:38:29.791783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.791682    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-oauth-serving-cert\") pod \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") "
Oct 30 20:38:29.791783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.791779    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-kube-api-access-4j4xz\" (UniqueName: \"kubernetes.io/projected/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-kube-api-access-4j4xz\") pod \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\" (UID: \"ef43ee01-2c9f-48e4-a1ac-9f21dc308579\") "
Oct 30 20:38:29.791968 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.791831    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-tmp" pod "ef43ee01-2c9f-48e4-a1ac-9f21dc308579" (UID: "ef43ee01-2c9f-48e4-a1ac-9f21dc308579"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:29.791968 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.791924    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-service-ca" pod "ef43ee01-2c9f-48e4-a1ac-9f21dc308579" (UID: "ef43ee01-2c9f-48e4-a1ac-9f21dc308579"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:29.792163 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.792137    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-oauth-serving-cert" pod "ef43ee01-2c9f-48e4-a1ac-9f21dc308579" (UID: "ef43ee01-2c9f-48e4-a1ac-9f21dc308579"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:29.792163 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.792152    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-config" pod "ef43ee01-2c9f-48e4-a1ac-9f21dc308579" (UID: "ef43ee01-2c9f-48e4-a1ac-9f21dc308579"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:29.792273 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.792191    4810 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:29.792273 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.792210    4810 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-tmp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:29.793842 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.793811    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-serving-cert" pod "ef43ee01-2c9f-48e4-a1ac-9f21dc308579" (UID: "ef43ee01-2c9f-48e4-a1ac-9f21dc308579"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:29.793956 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.793871    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-kube-api-access-4j4xz" pod "ef43ee01-2c9f-48e4-a1ac-9f21dc308579" (UID: "ef43ee01-2c9f-48e4-a1ac-9f21dc308579"). InnerVolumeSpecName "kube-api-access-4j4xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:29.793956 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.793904    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-oauth-config" pod "ef43ee01-2c9f-48e4-a1ac-9f21dc308579" (UID: "ef43ee01-2c9f-48e4-a1ac-9f21dc308579"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:29.893054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.893026    4810 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-config\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:29.893054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.893050    4810 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-serving-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:29.893195 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.893060    4810 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-oauth-serving-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:29.893195 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.893069    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j4xz\" (UniqueName: \"kubernetes.io/projected/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-kube-api-access-4j4xz\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:29.893195 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:29.893079    4810 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef43ee01-2c9f-48e4-a1ac-9f21dc308579-console-oauth-config\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:30.413809 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:30.413776    4810 generic.go:356] "Generic (PLEG): container finished" podID="ef43ee01-2c9f-48e4-a1ac-9f21dc308579" containerID="c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09" exitCode=0
Oct 30 20:38:30.414243 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:30.413832    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c4b849c-pg2d4"
Oct 30 20:38:30.414243 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:30.413843    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c4b849c-pg2d4" event={"ID":"ef43ee01-2c9f-48e4-a1ac-9f21dc308579","Type":"ContainerDied","Data":"c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09"}
Oct 30 20:38:30.414243 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:30.413912    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c4b849c-pg2d4" event={"ID":"ef43ee01-2c9f-48e4-a1ac-9f21dc308579","Type":"ContainerDied","Data":"ccb8a686c2c660e86f6e72f6552c7ee377b01f8c818f6f42c22d2379f975a6f0"}
Oct 30 20:38:30.414243 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:30.413928    4810 scope.go:122] "RemoveContainer" containerID="c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09"
Oct 30 20:38:30.423110 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:30.423095    4810 scope.go:122] "RemoveContainer" containerID="c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09"
Oct 30 20:38:30.423386 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:30.423360    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09\": container with ID starting with c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09 not found: ID does not exist" containerID="c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09"
Oct 30 20:38:30.423450 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:30.423399    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09"} err="failed to get container status \"c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09\": rpc error: code = NotFound desc = could not find container \"c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09\": container with ID starting with c4a4bee4a13adaedc1573a0972db9d2dddceef415a000ef2c6cb5652a3c65e09 not found: ID does not exist"
Oct 30 20:38:30.439914 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:30.439894    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857c4b849c-pg2d4"]
Oct 30 20:38:30.444356 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:30.444333    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-857c4b849c-pg2d4"]
Oct 30 20:38:31.660700 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:31.660666    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef43ee01-2c9f-48e4-a1ac-9f21dc308579" path="/var/lib/kubelet/pods/ef43ee01-2c9f-48e4-a1ac-9f21dc308579/volumes"
Oct 30 20:38:36.735937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:36.735901    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:36.752233 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:36.752207    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:37.464229 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:37.464206    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:40.655038 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:40.655006    4810 scope.go:122] "RemoveContainer" containerID="79ac9a599cf3242c80b15d85d7c35581883bed2757c24df1d5a15483a7b2e57b"
Oct 30 20:38:41.187985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.187952    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-x48bh-vgjns"]
Oct 30 20:38:41.188166 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.188146    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-x48bh-vgjns" podUID="3f567eec-9295-4e3e-bf6f-ec0e187eb3ed" containerName="insights-gathering" containerID="cri-o://87f7a8d97c0a0c3a0fccf8c2d968bdfb22862bb8d4e92f7227baa0f9f58c4adb" gracePeriod=30
Oct 30 20:38:41.469918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.469886    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rbf5z" event={"ID":"b3e2fc05-6adb-4c94-a124-aca4483a4e13","Type":"ContainerStarted","Data":"ff977dd5d91be823ead965334ddb2e97f250af4c04299b2c9ab078d923869f8a"}
Oct 30 20:38:41.471118 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.471096    4810 generic.go:356] "Generic (PLEG): container finished" podID="3f567eec-9295-4e3e-bf6f-ec0e187eb3ed" containerID="87f7a8d97c0a0c3a0fccf8c2d968bdfb22862bb8d4e92f7227baa0f9f58c4adb" exitCode=2
Oct 30 20:38:41.471220 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.471127    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x48bh-vgjns" event={"ID":"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed","Type":"ContainerDied","Data":"87f7a8d97c0a0c3a0fccf8c2d968bdfb22862bb8d4e92f7227baa0f9f58c4adb"}
Oct 30 20:38:41.471220 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.471157    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x48bh-vgjns" event={"ID":"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed","Type":"ContainerDied","Data":"be05cd365cd43efd11b6996141c40bc99ddd29342ba9f9aa7543396767fd386c"}
Oct 30 20:38:41.471220 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.471171    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be05cd365cd43efd11b6996141c40bc99ddd29342ba9f9aa7543396767fd386c"
Oct 30 20:38:41.490447 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.490427    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:38:41.612894 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.612842    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-kube-api-access-96blw\" (UniqueName: \"kubernetes.io/projected/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-kube-api-access-96blw\") pod \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") "
Oct 30 20:38:41.613037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.612922    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-service-ca-bundle\") pod \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") "
Oct 30 20:38:41.613037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.612965    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-trusted-ca-bundle\") pod \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") "
Oct 30 20:38:41.613037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.612989    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-archives-path\") pod \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\" (UID: \"3f567eec-9295-4e3e-bf6f-ec0e187eb3ed\") "
Oct 30 20:38:41.613385 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.613261    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-service-ca-bundle" pod "3f567eec-9295-4e3e-bf6f-ec0e187eb3ed" (UID: "3f567eec-9295-4e3e-bf6f-ec0e187eb3ed"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:41.613519 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.613404    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-trusted-ca-bundle" pod "3f567eec-9295-4e3e-bf6f-ec0e187eb3ed" (UID: "3f567eec-9295-4e3e-bf6f-ec0e187eb3ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:41.613519 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.613407    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-archives-path" pod "3f567eec-9295-4e3e-bf6f-ec0e187eb3ed" (UID: "3f567eec-9295-4e3e-bf6f-ec0e187eb3ed"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:41.614979 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.614957    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-kube-api-access-96blw" pod "3f567eec-9295-4e3e-bf6f-ec0e187eb3ed" (UID: "3f567eec-9295-4e3e-bf6f-ec0e187eb3ed"). InnerVolumeSpecName "kube-api-access-96blw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:41.714291 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.714235    4810 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-trusted-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:41.714291 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.714259    4810 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-archives-path\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:41.714291 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.714269    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96blw\" (UniqueName: \"kubernetes.io/projected/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-kube-api-access-96blw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:41.714291 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:41.714278    4810 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed-service-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:42.482539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:42.477339    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x48bh-vgjns"
Oct 30 20:38:42.497035 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:42.497009    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-x48bh-vgjns"]
Oct 30 20:38:42.503691 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:42.503669    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-x48bh-vgjns"]
Oct 30 20:38:43.483472 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:43.483441    4810 generic.go:356] "Generic (PLEG): container finished" podID="b4376a74-205b-45a1-b005-c59fae4f4201" containerID="191465458b3c2d97e34b6a8014d8204c0eda55bbdb2f5c4475a210695df9a72a" exitCode=0
Oct 30 20:38:43.483815 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:43.483509    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd" event={"ID":"b4376a74-205b-45a1-b005-c59fae4f4201","Type":"ContainerDied","Data":"191465458b3c2d97e34b6a8014d8204c0eda55bbdb2f5c4475a210695df9a72a"}
Oct 30 20:38:43.483815 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:43.483810    4810 scope.go:122] "RemoveContainer" containerID="191465458b3c2d97e34b6a8014d8204c0eda55bbdb2f5c4475a210695df9a72a"
Oct 30 20:38:43.661413 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:43.661381    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f567eec-9295-4e3e-bf6f-ec0e187eb3ed" path="/var/lib/kubelet/pods/3f567eec-9295-4e3e-bf6f-ec0e187eb3ed/volumes"
Oct 30 20:38:44.489994 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:44.489958    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-75fxd" event={"ID":"b4376a74-205b-45a1-b005-c59fae4f4201","Type":"ContainerStarted","Data":"735de97799833f78b883b5da4368165f7c17406d685eb13f1800a7a89e3f0931"}
Oct 30 20:38:48.134974 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.134939    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:38:48.135423 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.135247    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="alertmanager" containerID="cri-o://332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e" gracePeriod=120
Oct 30 20:38:48.135423 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.135313    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy-metric" containerID="cri-o://d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433" gracePeriod=120
Oct 30 20:38:48.135423 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.135341    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy-web" containerID="cri-o://9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd" gracePeriod=120
Oct 30 20:38:48.135423 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.135360    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="prom-label-proxy" containerID="cri-o://5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1" gracePeriod=120
Oct 30 20:38:48.135638 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.135426    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy" containerID="cri-o://cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2" gracePeriod=120
Oct 30 20:38:48.135638 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.135532    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="config-reloader" containerID="cri-o://a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1" gracePeriod=120
Oct 30 20:38:48.406601 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.406576    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:48.515005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.514973    4810 generic.go:356] "Generic (PLEG): container finished" podID="8a82d659-a274-4c7b-9935-5004f715b456" containerID="5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1" exitCode=0
Oct 30 20:38:48.515005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.514999    4810 generic.go:356] "Generic (PLEG): container finished" podID="8a82d659-a274-4c7b-9935-5004f715b456" containerID="d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433" exitCode=0
Oct 30 20:38:48.515005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515007    4810 generic.go:356] "Generic (PLEG): container finished" podID="8a82d659-a274-4c7b-9935-5004f715b456" containerID="cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2" exitCode=0
Oct 30 20:38:48.515005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515015    4810 generic.go:356] "Generic (PLEG): container finished" podID="8a82d659-a274-4c7b-9935-5004f715b456" containerID="9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd" exitCode=0
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515020    4810 generic.go:356] "Generic (PLEG): container finished" podID="8a82d659-a274-4c7b-9935-5004f715b456" containerID="a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1" exitCode=0
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515025    4810 generic.go:356] "Generic (PLEG): container finished" podID="8a82d659-a274-4c7b-9935-5004f715b456" containerID="332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e" exitCode=0
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515048    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerDied","Data":"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"}
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515062    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515123    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerDied","Data":"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"}
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515135    4810 scope.go:122] "RemoveContainer" containerID="5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515156    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerDied","Data":"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"}
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515187    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerDied","Data":"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"}
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515216    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerDied","Data":"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"}
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515242    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerDied","Data":"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"}
Oct 30 20:38:48.515344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.515279    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a82d659-a274-4c7b-9935-5004f715b456","Type":"ContainerDied","Data":"b0443d9e3499471e18cb268189df6499d83fdee019fafd73002c4c2b5c1bc776"}
Oct 30 20:38:48.523668 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.523652    4810 scope.go:122] "RemoveContainer" containerID="d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"
Oct 30 20:38:48.530888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.530872    4810 scope.go:122] "RemoveContainer" containerID="cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"
Oct 30 20:38:48.537936 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.537920    4810 scope.go:122] "RemoveContainer" containerID="9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"
Oct 30 20:38:48.544778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.544763    4810 scope.go:122] "RemoveContainer" containerID="a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"
Oct 30 20:38:48.551585 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.551567    4810 scope.go:122] "RemoveContainer" containerID="332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"
Oct 30 20:38:48.558745 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.558728    4810 scope.go:122] "RemoveContainer" containerID="0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"
Oct 30 20:38:48.565830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.565815    4810 scope.go:122] "RemoveContainer" containerID="5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"
Oct 30 20:38:48.566133 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:48.566110    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": container with ID starting with 5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1 not found: ID does not exist" containerID="5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"
Oct 30 20:38:48.566208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.566143    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"} err="failed to get container status \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": rpc error: code = NotFound desc = could not find container \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": container with ID starting with 5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1 not found: ID does not exist"
Oct 30 20:38:48.566208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.566165    4810 scope.go:122] "RemoveContainer" containerID="d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"
Oct 30 20:38:48.566404 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:48.566384    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": container with ID starting with d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433 not found: ID does not exist" containerID="d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"
Oct 30 20:38:48.566518 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.566408    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"} err="failed to get container status \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": rpc error: code = NotFound desc = could not find container \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": container with ID starting with d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433 not found: ID does not exist"
Oct 30 20:38:48.566518 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.566423    4810 scope.go:122] "RemoveContainer" containerID="cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"
Oct 30 20:38:48.566696 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:48.566675    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": container with ID starting with cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2 not found: ID does not exist" containerID="cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"
Oct 30 20:38:48.566751 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.566704    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"} err="failed to get container status \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": rpc error: code = NotFound desc = could not find container \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": container with ID starting with cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2 not found: ID does not exist"
Oct 30 20:38:48.566751 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.566723    4810 scope.go:122] "RemoveContainer" containerID="9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"
Oct 30 20:38:48.567831 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:48.567811    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": container with ID starting with 9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd not found: ID does not exist" containerID="9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"
Oct 30 20:38:48.567939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.567838    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"} err="failed to get container status \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": rpc error: code = NotFound desc = could not find container \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": container with ID starting with 9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd not found: ID does not exist"
Oct 30 20:38:48.567939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.567886    4810 scope.go:122] "RemoveContainer" containerID="a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"
Oct 30 20:38:48.568186 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:48.568168    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": container with ID starting with a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1 not found: ID does not exist" containerID="a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"
Oct 30 20:38:48.568232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.568191    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"} err="failed to get container status \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": rpc error: code = NotFound desc = could not find container \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": container with ID starting with a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1 not found: ID does not exist"
Oct 30 20:38:48.568232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.568205    4810 scope.go:122] "RemoveContainer" containerID="332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"
Oct 30 20:38:48.568450 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:48.568434    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": container with ID starting with 332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e not found: ID does not exist" containerID="332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"
Oct 30 20:38:48.568515 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.568473    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"} err="failed to get container status \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": rpc error: code = NotFound desc = could not find container \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": container with ID starting with 332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e not found: ID does not exist"
Oct 30 20:38:48.568515 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.568494    4810 scope.go:122] "RemoveContainer" containerID="0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"
Oct 30 20:38:48.568720 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:48.568701    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": container with ID starting with 0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0 not found: ID does not exist" containerID="0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"
Oct 30 20:38:48.568820 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.568723    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"} err="failed to get container status \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": rpc error: code = NotFound desc = could not find container \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": container with ID starting with 0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0 not found: ID does not exist"
Oct 30 20:38:48.568820 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.568741    4810 scope.go:122] "RemoveContainer" containerID="5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"
Oct 30 20:38:48.569022 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.569001    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"} err="failed to get container status \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": rpc error: code = NotFound desc = could not find container \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": container with ID starting with 5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1 not found: ID does not exist"
Oct 30 20:38:48.569072 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.569023    4810 scope.go:122] "RemoveContainer" containerID="d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"
Oct 30 20:38:48.569255 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.569236    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"} err="failed to get container status \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": rpc error: code = NotFound desc = could not find container \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": container with ID starting with d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433 not found: ID does not exist"
Oct 30 20:38:48.569327 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.569256    4810 scope.go:122] "RemoveContainer" containerID="cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"
Oct 30 20:38:48.569469 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.569453    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"} err="failed to get container status \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": rpc error: code = NotFound desc = could not find container \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": container with ID starting with cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2 not found: ID does not exist"
Oct 30 20:38:48.569535 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.569470    4810 scope.go:122] "RemoveContainer" containerID="9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"
Oct 30 20:38:48.569669 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.569653    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"} err="failed to get container status \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": rpc error: code = NotFound desc = could not find container \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": container with ID starting with 9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd not found: ID does not exist"
Oct 30 20:38:48.569731 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.569671    4810 scope.go:122] "RemoveContainer" containerID="a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"
Oct 30 20:38:48.569911 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.569888    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"} err="failed to get container status \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": rpc error: code = NotFound desc = could not find container \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": container with ID starting with a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1 not found: ID does not exist"
Oct 30 20:38:48.569911 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.569908    4810 scope.go:122] "RemoveContainer" containerID="332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"
Oct 30 20:38:48.570126 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.570110    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"} err="failed to get container status \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": rpc error: code = NotFound desc = could not find container \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": container with ID starting with 332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e not found: ID does not exist"
Oct 30 20:38:48.570172 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.570125    4810 scope.go:122] "RemoveContainer" containerID="0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"
Oct 30 20:38:48.570338 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.570323    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"} err="failed to get container status \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": rpc error: code = NotFound desc = could not find container \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": container with ID starting with 0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0 not found: ID does not exist"
Oct 30 20:38:48.570338 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.570336    4810 scope.go:122] "RemoveContainer" containerID="5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"
Oct 30 20:38:48.570563 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.570544    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"} err="failed to get container status \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": rpc error: code = NotFound desc = could not find container \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": container with ID starting with 5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1 not found: ID does not exist"
Oct 30 20:38:48.570613 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.570564    4810 scope.go:122] "RemoveContainer" containerID="d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"
Oct 30 20:38:48.570775 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.570757    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"} err="failed to get container status \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": rpc error: code = NotFound desc = could not find container \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": container with ID starting with d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433 not found: ID does not exist"
Oct 30 20:38:48.570833 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.570775    4810 scope.go:122] "RemoveContainer" containerID="cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"
Oct 30 20:38:48.570993 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.570976    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"} err="failed to get container status \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": rpc error: code = NotFound desc = could not find container \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": container with ID starting with cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2 not found: ID does not exist"
Oct 30 20:38:48.571049 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.570993    4810 scope.go:122] "RemoveContainer" containerID="9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"
Oct 30 20:38:48.571210 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.571192    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"} err="failed to get container status \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": rpc error: code = NotFound desc = could not find container \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": container with ID starting with 9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd not found: ID does not exist"
Oct 30 20:38:48.571263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.571210    4810 scope.go:122] "RemoveContainer" containerID="a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"
Oct 30 20:38:48.571414 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.571395    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"} err="failed to get container status \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": rpc error: code = NotFound desc = could not find container \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": container with ID starting with a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1 not found: ID does not exist"
Oct 30 20:38:48.571468 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.571414    4810 scope.go:122] "RemoveContainer" containerID="332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"
Oct 30 20:38:48.571609 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.571592    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"} err="failed to get container status \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": rpc error: code = NotFound desc = could not find container \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": container with ID starting with 332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e not found: ID does not exist"
Oct 30 20:38:48.571652 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.571609    4810 scope.go:122] "RemoveContainer" containerID="0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"
Oct 30 20:38:48.571813 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.571797    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"} err="failed to get container status \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": rpc error: code = NotFound desc = could not find container \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": container with ID starting with 0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0 not found: ID does not exist"
Oct 30 20:38:48.571813 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.571812    4810 scope.go:122] "RemoveContainer" containerID="5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"
Oct 30 20:38:48.572024 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572007    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"} err="failed to get container status \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": rpc error: code = NotFound desc = could not find container \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": container with ID starting with 5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1 not found: ID does not exist"
Oct 30 20:38:48.572024 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572024    4810 scope.go:122] "RemoveContainer" containerID="d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"
Oct 30 20:38:48.572225 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572210    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"} err="failed to get container status \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": rpc error: code = NotFound desc = could not find container \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": container with ID starting with d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433 not found: ID does not exist"
Oct 30 20:38:48.572270 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572225    4810 scope.go:122] "RemoveContainer" containerID="cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"
Oct 30 20:38:48.572404 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572390    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"} err="failed to get container status \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": rpc error: code = NotFound desc = could not find container \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": container with ID starting with cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2 not found: ID does not exist"
Oct 30 20:38:48.572445 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572405    4810 scope.go:122] "RemoveContainer" containerID="9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"
Oct 30 20:38:48.572619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572603    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"} err="failed to get container status \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": rpc error: code = NotFound desc = could not find container \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": container with ID starting with 9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd not found: ID does not exist"
Oct 30 20:38:48.572656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572619    4810 scope.go:122] "RemoveContainer" containerID="a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"
Oct 30 20:38:48.572807 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572791    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"} err="failed to get container status \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": rpc error: code = NotFound desc = could not find container \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": container with ID starting with a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1 not found: ID does not exist"
Oct 30 20:38:48.572849 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572807    4810 scope.go:122] "RemoveContainer" containerID="332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"
Oct 30 20:38:48.573014 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.572997    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"} err="failed to get container status \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": rpc error: code = NotFound desc = could not find container \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": container with ID starting with 332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e not found: ID does not exist"
Oct 30 20:38:48.573054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.573015    4810 scope.go:122] "RemoveContainer" containerID="0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"
Oct 30 20:38:48.573229 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.573211    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"} err="failed to get container status \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": rpc error: code = NotFound desc = could not find container \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": container with ID starting with 0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0 not found: ID does not exist"
Oct 30 20:38:48.573229 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.573228    4810 scope.go:122] "RemoveContainer" containerID="5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"
Oct 30 20:38:48.573436 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.573420    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"} err="failed to get container status \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": rpc error: code = NotFound desc = could not find container \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": container with ID starting with 5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1 not found: ID does not exist"
Oct 30 20:38:48.573491 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.573436    4810 scope.go:122] "RemoveContainer" containerID="d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"
Oct 30 20:38:48.573632 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.573616    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"} err="failed to get container status \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": rpc error: code = NotFound desc = could not find container \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": container with ID starting with d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433 not found: ID does not exist"
Oct 30 20:38:48.573632 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.573631    4810 scope.go:122] "RemoveContainer" containerID="cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"
Oct 30 20:38:48.573826 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.573811    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"} err="failed to get container status \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": rpc error: code = NotFound desc = could not find container \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": container with ID starting with cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2 not found: ID does not exist"
Oct 30 20:38:48.573826 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.573826    4810 scope.go:122] "RemoveContainer" containerID="9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"
Oct 30 20:38:48.574058 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.574042    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"} err="failed to get container status \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": rpc error: code = NotFound desc = could not find container \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": container with ID starting with 9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd not found: ID does not exist"
Oct 30 20:38:48.574058 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.574057    4810 scope.go:122] "RemoveContainer" containerID="a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"
Oct 30 20:38:48.574274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.574255    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"} err="failed to get container status \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": rpc error: code = NotFound desc = could not find container \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": container with ID starting with a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1 not found: ID does not exist"
Oct 30 20:38:48.574321 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.574274    4810 scope.go:122] "RemoveContainer" containerID="332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"
Oct 30 20:38:48.574493 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.574476    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"} err="failed to get container status \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": rpc error: code = NotFound desc = could not find container \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": container with ID starting with 332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e not found: ID does not exist"
Oct 30 20:38:48.574493 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.574492    4810 scope.go:122] "RemoveContainer" containerID="0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"
Oct 30 20:38:48.574710 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.574691    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"} err="failed to get container status \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": rpc error: code = NotFound desc = could not find container \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": container with ID starting with 0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0 not found: ID does not exist"
Oct 30 20:38:48.574748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.574710    4810 scope.go:122] "RemoveContainer" containerID="5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"
Oct 30 20:38:48.574955 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.574938    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1"} err="failed to get container status \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": rpc error: code = NotFound desc = could not find container \"5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1\": container with ID starting with 5127093875e755dce3afffe4c621251b2823345bbc1db862c5f1d50139acf8e1 not found: ID does not exist"
Oct 30 20:38:48.575003 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.574956    4810 scope.go:122] "RemoveContainer" containerID="d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"
Oct 30 20:38:48.575182 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.575166    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433"} err="failed to get container status \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": rpc error: code = NotFound desc = could not find container \"d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433\": container with ID starting with d454964a7949902d34a1f080f15783fa1b4ee0f58a6219979966b97c7d06c433 not found: ID does not exist"
Oct 30 20:38:48.575229 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.575183    4810 scope.go:122] "RemoveContainer" containerID="cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"
Oct 30 20:38:48.575403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.575386    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2"} err="failed to get container status \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": rpc error: code = NotFound desc = could not find container \"cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2\": container with ID starting with cc57d936a5d2e743b1a3b38e4dbc744fd0f4b7287626d999fe8e056ed1c0d8b2 not found: ID does not exist"
Oct 30 20:38:48.575447 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.575404    4810 scope.go:122] "RemoveContainer" containerID="9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"
Oct 30 20:38:48.575602 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.575586    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd"} err="failed to get container status \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": rpc error: code = NotFound desc = could not find container \"9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd\": container with ID starting with 9d85177b0e603b3c60e5ac2098ce98349e64d3360acc821a0282724eaedad6cd not found: ID does not exist"
Oct 30 20:38:48.575637 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.575602    4810 scope.go:122] "RemoveContainer" containerID="a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"
Oct 30 20:38:48.575820 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.575804    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1"} err="failed to get container status \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": rpc error: code = NotFound desc = could not find container \"a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1\": container with ID starting with a2c2970534ad826f33eb2bb1d55321a1d1bff5b110756f61bd02e144696e76b1 not found: ID does not exist"
Oct 30 20:38:48.575911 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.575820    4810 scope.go:122] "RemoveContainer" containerID="332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"
Oct 30 20:38:48.576022 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.576003    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e"} err="failed to get container status \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": rpc error: code = NotFound desc = could not find container \"332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e\": container with ID starting with 332e55073394b57e2bb9424cc6d37aa149f970cf2c17a6ab614b2c718d02de7e not found: ID does not exist"
Oct 30 20:38:48.576061 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.576023    4810 scope.go:122] "RemoveContainer" containerID="0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"
Oct 30 20:38:48.576222 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.576206    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0"} err="failed to get container status \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": rpc error: code = NotFound desc = could not find container \"0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0\": container with ID starting with 0d76fe19608385a52a3471bfe50aaf00aef8571138659ce8b8e799cdbe87b2b0 not found: ID does not exist"
Oct 30 20:38:48.577461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.577446    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-main-tls\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.577511 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.577485    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-web-config\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-web-config\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.577553 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.577507    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-metrics-client-ca\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.577592 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.577557    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.577627 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.577619    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-config-volume\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-config-volume\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.577678 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.577662    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-web\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.577829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.577772    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.577955 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.577929    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-metrics-client-ca" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:48.578236 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.578188    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:48.578384 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.577852    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-cluster-tls-config\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.578490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.578470    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-main-db\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.578558 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.578542    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.578619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.578574    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-config-out\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.578619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.578609    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-kube-api-access-cpbgm\" (UniqueName: \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-kube-api-access-cpbgm\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.578721 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.578635    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-tls-assets\" (UniqueName: \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-tls-assets\") pod \"8a82d659-a274-4c7b-9935-5004f715b456\" (UID: \"8a82d659-a274-4c7b-9935-5004f715b456\") "
Oct 30 20:38:48.578808 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.578784    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-main-db" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:48.579144 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.579119    4810 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-main-db\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.579261 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.579147    4810 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-metrics-client-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.579261 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.579164    4810 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a82d659-a274-4c7b-9935-5004f715b456-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.580078 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.580049    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-config-volume" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:48.580541 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.580519    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-metric" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:48.581379 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.581351    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-web" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:48.581550 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.581529    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:48.581935 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.581841    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-tls-assets" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:48.581935 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.581915    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-main-tls" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:48.582077 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.581937    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-config-out" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:48.582077 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.582005    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-kube-api-access-cpbgm" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "kube-api-access-cpbgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:48.586391 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.586370    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-cluster-tls-config" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:48.593558 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.593540    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-web-config" pod "8a82d659-a274-4c7b-9935-5004f715b456" (UID: "8a82d659-a274-4c7b-9935-5004f715b456"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:48.680337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.680278    4810 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.680337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.680300    4810 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.680337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.680310    4810 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-cluster-tls-config\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.680337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.680319    4810 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.680337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.680329    4810 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a82d659-a274-4c7b-9935-5004f715b456-config-out\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.680337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.680338    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpbgm\" (UniqueName: \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-kube-api-access-cpbgm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.680591 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.680348    4810 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a82d659-a274-4c7b-9935-5004f715b456-tls-assets\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.680591 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.680358    4810 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-main-tls\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.680591 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.680367    4810 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-web-config\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.680591 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.680375    4810 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a82d659-a274-4c7b-9935-5004f715b456-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:48.842789 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.842760    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:38:48.849352 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:48.849328    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:38:49.659625 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:49.659596    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a82d659-a274-4c7b-9935-5004f715b456" path="/var/lib/kubelet/pods/8a82d659-a274-4c7b-9935-5004f715b456/volumes"
Oct 30 20:38:52.397874 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.397816    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:38:52.398326 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.398213    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="prometheus" containerID="cri-o://700d5b7cbee1ac9997c4fcc5d0bb6bfa55a296ad47610c742fec83fac6a6def8" gracePeriod=600
Oct 30 20:38:52.398616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.398591    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://52afeedf8a175f32b712f288ff2dee37aca5b7ac5f6aa6b980516569de6e722b" gracePeriod=600
Oct 30 20:38:52.398717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.398693    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy" containerID="cri-o://b8e493632cf15885137f7774c7c9f5b0e3e90c1b49d73c14c665b3799110a0be" gracePeriod=600
Oct 30 20:38:52.398809 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.398793    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy-web" containerID="cri-o://9dbd5fceec594eebfa82fcb2449d989dd01fecc40416fbc2a56a431a8867a8bb" gracePeriod=600
Oct 30 20:38:52.398956 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.398905    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="config-reloader" containerID="cri-o://5e1941c360b1814b26d21f483399c75b9270f1040ad22b52a3378f7672df82c8" gracePeriod=600
Oct 30 20:38:52.399647 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.399057    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="thanos-sidecar" containerID="cri-o://0de84fa73f287378c1e61b1c1b279907eed8c5e6fb0e64eb619ee08f14b8fe22" gracePeriod=600
Oct 30 20:38:52.449259 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:52.449202    4810 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="700d5b7cbee1ac9997c4fcc5d0bb6bfa55a296ad47610c742fec83fac6a6def8" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 20:38:52.449407 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:52.449285    4810 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="700d5b7cbee1ac9997c4fcc5d0bb6bfa55a296ad47610c742fec83fac6a6def8" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 20:38:52.449407 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:52.449312    4810 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="700d5b7cbee1ac9997c4fcc5d0bb6bfa55a296ad47610c742fec83fac6a6def8" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 20:38:52.449407 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.449335    4810 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Oct 30 20:38:52.538277 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538240    4810 generic.go:356] "Generic (PLEG): container finished" podID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerID="52afeedf8a175f32b712f288ff2dee37aca5b7ac5f6aa6b980516569de6e722b" exitCode=0
Oct 30 20:38:52.538277 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538264    4810 generic.go:356] "Generic (PLEG): container finished" podID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerID="b8e493632cf15885137f7774c7c9f5b0e3e90c1b49d73c14c665b3799110a0be" exitCode=0
Oct 30 20:38:52.538277 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538273    4810 generic.go:356] "Generic (PLEG): container finished" podID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerID="9dbd5fceec594eebfa82fcb2449d989dd01fecc40416fbc2a56a431a8867a8bb" exitCode=0
Oct 30 20:38:52.538277 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538281    4810 generic.go:356] "Generic (PLEG): container finished" podID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerID="0de84fa73f287378c1e61b1c1b279907eed8c5e6fb0e64eb619ee08f14b8fe22" exitCode=0
Oct 30 20:38:52.538487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538289    4810 generic.go:356] "Generic (PLEG): container finished" podID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerID="5e1941c360b1814b26d21f483399c75b9270f1040ad22b52a3378f7672df82c8" exitCode=0
Oct 30 20:38:52.538487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538297    4810 generic.go:356] "Generic (PLEG): container finished" podID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerID="700d5b7cbee1ac9997c4fcc5d0bb6bfa55a296ad47610c742fec83fac6a6def8" exitCode=0
Oct 30 20:38:52.538487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538309    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerDied","Data":"52afeedf8a175f32b712f288ff2dee37aca5b7ac5f6aa6b980516569de6e722b"}
Oct 30 20:38:52.538487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538371    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerDied","Data":"b8e493632cf15885137f7774c7c9f5b0e3e90c1b49d73c14c665b3799110a0be"}
Oct 30 20:38:52.538487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538398    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerDied","Data":"9dbd5fceec594eebfa82fcb2449d989dd01fecc40416fbc2a56a431a8867a8bb"}
Oct 30 20:38:52.538487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538422    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerDied","Data":"0de84fa73f287378c1e61b1c1b279907eed8c5e6fb0e64eb619ee08f14b8fe22"}
Oct 30 20:38:52.538487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538446    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerDied","Data":"5e1941c360b1814b26d21f483399c75b9270f1040ad22b52a3378f7672df82c8"}
Oct 30 20:38:52.538487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.538468    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerDied","Data":"700d5b7cbee1ac9997c4fcc5d0bb6bfa55a296ad47610c742fec83fac6a6def8"}
Oct 30 20:38:52.683368 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.683344    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:52.818850 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.818814    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-1\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.818896    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-serving-certs-ca-bundle\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.818943    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.818977    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819013    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-0\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819051    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819090    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-grpc-tls\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819123    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-metrics-client-ca\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819165    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-kube-api-access-swr5m\" (UniqueName: \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-kube-api-access-swr5m\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819224    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config-out\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819222    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-1" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819323    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-tls\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819354    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-tls-assets\" (UniqueName: \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-tls-assets\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819372    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-serving-certs-ca-bundle" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819384    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-2\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819422    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-web-config\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-web-config\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819451    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-kubelet-serving-ca-bundle\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819479    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-db\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819518    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-trusted-ca-bundle\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.819580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819545    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-kube-rbac-proxy\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.820127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819584    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-metrics-client-certs\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.820127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819622    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-thanos-prometheus-http-client-file\") pod \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\" (UID: \"dd3e1b20-0cd4-4161-b4e0-0125f944439a\") "
Oct 30 20:38:52.820127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.819626    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-metrics-client-ca" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:52.820127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.820030    4810 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-metrics-client-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.820127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.820052    4810 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.820127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.820069    4810 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.820608 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.820588    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-trusted-ca-bundle" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:52.821209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.820917    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-0" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:52.821361 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.821198    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-db" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:52.822173 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.821907    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-thanos-sidecar-tls" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:52.822173 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.822002    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:52.822173 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.822108    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-grpc-tls" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:52.822412 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.822278    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config-out" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:52.822561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.822533    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-kube-rbac-proxy" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:52.822678 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.822593    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-2" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:52.822991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.822906    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:52.822991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.823085    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-kubelet-serving-ca-bundle" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:52.823423 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.823304    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-thanos-prometheus-http-client-file" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:52.823786 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.823759    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-tls" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:52.824082 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.824049    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-kube-api-access-swr5m" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "kube-api-access-swr5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:52.824421 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.824404    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-tls-assets" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:52.824633 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.824613    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-metrics-client-certs" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:52.836733 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.836710    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-web-config" pod "dd3e1b20-0cd4-4161-b4e0-0125f944439a" (UID: "dd3e1b20-0cd4-4161-b4e0-0125f944439a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:52.920810 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920758    4810 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.920810 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920801    4810 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.920810 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920813    4810 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.920810 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920823    4810 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920832    4810 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-grpc-tls\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920841    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swr5m\" (UniqueName: \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-kube-api-access-swr5m\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920850    4810 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-config-out\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920888    4810 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920897    4810 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dd3e1b20-0cd4-4161-b4e0-0125f944439a-tls-assets\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920907    4810 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920918    4810 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-web-config\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920928    4810 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920937    4810 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-k8s-db\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920945    4810 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd3e1b20-0cd4-4161-b4e0-0125f944439a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920954    4810 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-kube-rbac-proxy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920962    4810 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-secret-metrics-client-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:52.921116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:52.920971    4810 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dd3e1b20-0cd4-4161-b4e0-0125f944439a-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:53.546313 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.546284    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:53.546815 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.546285    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dd3e1b20-0cd4-4161-b4e0-0125f944439a","Type":"ContainerDied","Data":"819473c68daee4b359c0edd56c6fac75eb6915086b7c2d063398399ee070502e"}
Oct 30 20:38:53.546815 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.546467    4810 scope.go:122] "RemoveContainer" containerID="52afeedf8a175f32b712f288ff2dee37aca5b7ac5f6aa6b980516569de6e722b"
Oct 30 20:38:53.555843 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.555818    4810 scope.go:122] "RemoveContainer" containerID="b8e493632cf15885137f7774c7c9f5b0e3e90c1b49d73c14c665b3799110a0be"
Oct 30 20:38:53.563433 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.563418    4810 scope.go:122] "RemoveContainer" containerID="9dbd5fceec594eebfa82fcb2449d989dd01fecc40416fbc2a56a431a8867a8bb"
Oct 30 20:38:53.571202 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.571183    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:38:53.572200 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.572167    4810 scope.go:122] "RemoveContainer" containerID="0de84fa73f287378c1e61b1c1b279907eed8c5e6fb0e64eb619ee08f14b8fe22"
Oct 30 20:38:53.575852 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.575829    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:38:53.582403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.582386    4810 scope.go:122] "RemoveContainer" containerID="5e1941c360b1814b26d21f483399c75b9270f1040ad22b52a3378f7672df82c8"
Oct 30 20:38:53.590017 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.589999    4810 scope.go:122] "RemoveContainer" containerID="700d5b7cbee1ac9997c4fcc5d0bb6bfa55a296ad47610c742fec83fac6a6def8"
Oct 30 20:38:53.597688 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.597657    4810 scope.go:122] "RemoveContainer" containerID="3dcf20713c2b45729e55284bd77f1484a021952d8dab8056c8ad557aed12a42a"
Oct 30 20:38:53.659600 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:53.659575    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" path="/var/lib/kubelet/pods/dd3e1b20-0cd4-4161-b4e0-0125f944439a/volumes"
Oct 30 20:38:55.155012 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:38:55.154973    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:38:56.166773 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:38:56.166740    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9698fc4f-8fzsh"]
Oct 30 20:39:21.187325 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.187271    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d9698fc4f-8fzsh" podUID="ae527ee9-5da3-4627-8176-5226765ad684" containerName="console" containerID="cri-o://c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae" gracePeriod=15
Oct 30 20:39:21.461023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.460998    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:39:21.563052 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563025    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-oauth-serving-cert\") pod \"ae527ee9-5da3-4627-8176-5226765ad684\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") "
Oct 30 20:39:21.563186 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563064    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae527ee9-5da3-4627-8176-5226765ad684-kube-api-access-qqm54\" (UniqueName: \"kubernetes.io/projected/ae527ee9-5da3-4627-8176-5226765ad684-kube-api-access-qqm54\") pod \"ae527ee9-5da3-4627-8176-5226765ad684\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") "
Oct 30 20:39:21.563186 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563110    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-service-ca\") pod \"ae527ee9-5da3-4627-8176-5226765ad684\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") "
Oct 30 20:39:21.563186 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563173    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-console-config\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-console-config\") pod \"ae527ee9-5da3-4627-8176-5226765ad684\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") "
Oct 30 20:39:21.563298 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563205    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-serving-cert\") pod \"ae527ee9-5da3-4627-8176-5226765ad684\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") "
Oct 30 20:39:21.563298 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563243    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae527ee9-5da3-4627-8176-5226765ad684-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae527ee9-5da3-4627-8176-5226765ad684-tmp\") pod \"ae527ee9-5da3-4627-8176-5226765ad684\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") "
Oct 30 20:39:21.563377 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563297    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-oauth-config\") pod \"ae527ee9-5da3-4627-8176-5226765ad684\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") "
Oct 30 20:39:21.563377 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563344    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-trusted-ca-bundle\") pod \"ae527ee9-5da3-4627-8176-5226765ad684\" (UID: \"ae527ee9-5da3-4627-8176-5226765ad684\") "
Oct 30 20:39:21.563558 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563532    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-oauth-serving-cert" pod "ae527ee9-5da3-4627-8176-5226765ad684" (UID: "ae527ee9-5da3-4627-8176-5226765ad684"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:21.563639 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563572    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae527ee9-5da3-4627-8176-5226765ad684-tmp" pod "ae527ee9-5da3-4627-8176-5226765ad684" (UID: "ae527ee9-5da3-4627-8176-5226765ad684"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:21.563639 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563625    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-service-ca" pod "ae527ee9-5da3-4627-8176-5226765ad684" (UID: "ae527ee9-5da3-4627-8176-5226765ad684"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:21.563759 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563631    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-console-config" pod "ae527ee9-5da3-4627-8176-5226765ad684" (UID: "ae527ee9-5da3-4627-8176-5226765ad684"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:21.563897 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.563878    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-trusted-ca-bundle" pod "ae527ee9-5da3-4627-8176-5226765ad684" (UID: "ae527ee9-5da3-4627-8176-5226765ad684"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:21.565424 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.565401    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-serving-cert" pod "ae527ee9-5da3-4627-8176-5226765ad684" (UID: "ae527ee9-5da3-4627-8176-5226765ad684"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:21.565524 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.565426    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae527ee9-5da3-4627-8176-5226765ad684-kube-api-access-qqm54" pod "ae527ee9-5da3-4627-8176-5226765ad684" (UID: "ae527ee9-5da3-4627-8176-5226765ad684"). InnerVolumeSpecName "kube-api-access-qqm54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:21.565524 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.565443    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-oauth-config" pod "ae527ee9-5da3-4627-8176-5226765ad684" (UID: "ae527ee9-5da3-4627-8176-5226765ad684"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:21.664523 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.664502    4810 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-serving-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.664523 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.664524    4810 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae527ee9-5da3-4627-8176-5226765ad684-tmp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.664659 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.664534    4810 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae527ee9-5da3-4627-8176-5226765ad684-console-oauth-config\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.664659 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.664543    4810 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-trusted-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.664659 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.664551    4810 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-oauth-serving-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.664659 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.664561    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqm54\" (UniqueName: \"kubernetes.io/projected/ae527ee9-5da3-4627-8176-5226765ad684-kube-api-access-qqm54\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.664659 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.664571    4810 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.664659 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.664579    4810 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae527ee9-5da3-4627-8176-5226765ad684-console-config\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.672900 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.672848    4810 generic.go:356] "Generic (PLEG): container finished" podID="ae527ee9-5da3-4627-8176-5226765ad684" containerID="c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae" exitCode=0
Oct 30 20:39:21.672982 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.672909    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9698fc4f-8fzsh"
Oct 30 20:39:21.672982 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.672931    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9698fc4f-8fzsh" event={"ID":"ae527ee9-5da3-4627-8176-5226765ad684","Type":"ContainerDied","Data":"c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae"}
Oct 30 20:39:21.673053 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.672980    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9698fc4f-8fzsh" event={"ID":"ae527ee9-5da3-4627-8176-5226765ad684","Type":"ContainerDied","Data":"de0651acf3d4acbeddd6a247eadec240562ea326844154f484dbb14a2e603d25"}
Oct 30 20:39:21.673053 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.673001    4810 scope.go:122] "RemoveContainer" containerID="c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae"
Oct 30 20:39:21.681271 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.681253    4810 scope.go:122] "RemoveContainer" containerID="c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae"
Oct 30 20:39:21.681587 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:39:21.681570    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae\": container with ID starting with c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae not found: ID does not exist" containerID="c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae"
Oct 30 20:39:21.681636 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.681594    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae"} err="failed to get container status \"c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae\": rpc error: code = NotFound desc = could not find container \"c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae\": container with ID starting with c0f7a15ee5b1815643cfc5875a2d3a6edc8953bfd56bb0db33443196be88b4ae not found: ID does not exist"
Oct 30 20:39:21.692571 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.692523    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9698fc4f-8fzsh"]
Oct 30 20:39:21.696617 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:21.696594    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d9698fc4f-8fzsh"]
Oct 30 20:39:23.661429 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:39:23.661395    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae527ee9-5da3-4627-8176-5226765ad684" path="/var/lib/kubelet/pods/ae527ee9-5da3-4627-8176-5226765ad684/volumes"
Oct 30 20:39:25.154371 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:39:25.154295    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:39:55.155108 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:39:55.155074    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:40:25.154883 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:40:25.154828    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:40:55.153668 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:40:55.153587    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:41:25.155125 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:41:25.155087    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:41:55.154605 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:41:55.154570    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:41:55.331938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.331906    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"]
Oct 30 20:41:55.332429 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332410    4810 cpu_manager.go:414] "containerMap: removing container" podUID="204ba748-a514-4772-9fba-dfafd3eddcf7" containerName="registry"
Oct 30 20:41:55.332478 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332433    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204ba748-a514-4772-9fba-dfafd3eddcf7" containerName="registry"
Oct 30 20:41:55.332478 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332447    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="config-reloader"
Oct 30 20:41:55.332478 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332454    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="config-reloader"
Oct 30 20:41:55.332478 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332464    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy-thanos"
Oct 30 20:41:55.332478 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332473    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy-thanos"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332490    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy-metric"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332497    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy-metric"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332522    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3f567eec-9295-4e3e-bf6f-ec0e187eb3ed" containerName="insights-gathering"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332529    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f567eec-9295-4e3e-bf6f-ec0e187eb3ed" containerName="insights-gathering"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332545    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="init-config-reloader"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332552    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="init-config-reloader"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332567    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="config-reloader"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332577    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="config-reloader"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332593    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ef43ee01-2c9f-48e4-a1ac-9f21dc308579" containerName="console"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332602    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef43ee01-2c9f-48e4-a1ac-9f21dc308579" containerName="console"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332614    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="alertmanager"
Oct 30 20:41:55.332623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332623    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="alertmanager"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332636    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="prom-label-proxy"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332644    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="prom-label-proxy"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332652    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy-web"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332659    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy-web"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332673    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="init-config-reloader"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332682    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="init-config-reloader"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332700    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="prometheus"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332707    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="prometheus"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332719    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy-web"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332726    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy-web"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332738    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332745    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332758    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ae527ee9-5da3-4627-8176-5226765ad684" containerName="console"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332765    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae527ee9-5da3-4627-8176-5226765ad684" containerName="console"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332775    4810 cpu_manager.go:414] "containerMap: removing container" podUID="2d2edfa6-676f-47ab-af10-37be9080c821" containerName="insights-gathering"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332782    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2edfa6-676f-47ab-af10-37be9080c821" containerName="insights-gathering"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332794    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4dd510fb-f40e-401c-845f-23b9d750dc68" containerName="registry"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332801    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd510fb-f40e-401c-845f-23b9d750dc68" containerName="registry"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332810    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3c202457-a065-42bd-b310-813616b4d8ef" containerName="console"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332817    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c202457-a065-42bd-b310-813616b4d8ef" containerName="console"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332827    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332834    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332842    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="thanos-sidecar"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332849    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="thanos-sidecar"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332966    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f567eec-9295-4e3e-bf6f-ec0e187eb3ed" containerName="insights-gathering"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332979    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy-web"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332988    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="config-reloader"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.332999    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef43ee01-2c9f-48e4-a1ac-9f21dc308579" containerName="console"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333005    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="alertmanager"
Oct 30 20:41:55.333002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333013    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="prom-label-proxy"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333023    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="prometheus"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333031    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy-web"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333041    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333050    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae527ee9-5da3-4627-8176-5226765ad684" containerName="console"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333057    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2edfa6-676f-47ab-af10-37be9080c821" containerName="insights-gathering"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333066    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd510fb-f40e-401c-845f-23b9d750dc68" containerName="registry"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333074    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c202457-a065-42bd-b310-813616b4d8ef" containerName="console"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333082    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333090    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="thanos-sidecar"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333099    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="204ba748-a514-4772-9fba-dfafd3eddcf7" containerName="registry"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333107    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="config-reloader"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333113    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3e1b20-0cd4-4161-b4e0-0125f944439a" containerName="kube-rbac-proxy-thanos"
Oct 30 20:41:55.333913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.333123    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a82d659-a274-4c7b-9935-5004f715b456" containerName="kube-rbac-proxy-metric"
Oct 30 20:41:55.336738 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.336719    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.380376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.380343    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"]
Oct 30 20:41:55.413198 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.413144    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 20:41:55.413682 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.413632    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-x2mvh\""
Oct 30 20:41:55.413932 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.413912    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 20:41:55.414073 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.414055    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 20:41:55.414169 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.414094    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 20:41:55.414208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.414192    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 20:41:55.432712 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.432691    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-metrics-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.432801 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.432719    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4s59\" (UniqueName: \"kubernetes.io/projected/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-kube-api-access-q4s59\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.432801 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.432749    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.432893 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.432853    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-webhook-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.432930 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.432901    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-manifests\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.432967 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.432934    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-tmp\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.534255 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.534223    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-metrics-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.534398 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.534257    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4s59\" (UniqueName: \"kubernetes.io/projected/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-kube-api-access-q4s59\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.534398 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.534300    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.534398 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.534345    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-webhook-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.534398 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.534384    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-manifests\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.534615 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.534425    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-tmp\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.534823 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.534790    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-manifests\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.536776 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.536751    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-metrics-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.536898 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.536797    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-webhook-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.536987 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.536971    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-tmp\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.537201 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.537183    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.548431 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.548404    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4s59\" (UniqueName: \"kubernetes.io/projected/a2b7525a-f37d-48a2-bd4d-e763c8b155dc-kube-api-access-q4s59\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-pz44x\" (UID: \"a2b7525a-f37d-48a2-bd4d-e763c8b155dc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.652259 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.652232    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:41:55.965983 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:55.965956    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"]
Oct 30 20:41:55.967417 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:41:55.967392    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b7525a_f37d_48a2_bd4d_e763c8b155dc.slice/crio-79db38c6b8ffbe904a6f259ac12b4223957b9b0603740c4f175522664373da40 WatchSource:0}: Error finding container 79db38c6b8ffbe904a6f259ac12b4223957b9b0603740c4f175522664373da40: Status 404 returned error can't find the container with id 79db38c6b8ffbe904a6f259ac12b4223957b9b0603740c4f175522664373da40
Oct 30 20:41:56.341805 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:56.341731    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x" event={"ID":"a2b7525a-f37d-48a2-bd4d-e763c8b155dc","Type":"ContainerStarted","Data":"79db38c6b8ffbe904a6f259ac12b4223957b9b0603740c4f175522664373da40"}
Oct 30 20:41:59.360308 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:41:59.360218    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x" event={"ID":"a2b7525a-f37d-48a2-bd4d-e763c8b155dc","Type":"ContainerStarted","Data":"4ad850f4634da0e2e217c2cb1a26088ad7d6bcab90e162d6eb724306d1b4f6bf"}
Oct 30 20:42:00.365834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:00.365800    4810 generic.go:356] "Generic (PLEG): container finished" podID="a2b7525a-f37d-48a2-bd4d-e763c8b155dc" containerID="4ad850f4634da0e2e217c2cb1a26088ad7d6bcab90e162d6eb724306d1b4f6bf" exitCode=0
Oct 30 20:42:00.366365 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:00.365844    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x" event={"ID":"a2b7525a-f37d-48a2-bd4d-e763c8b155dc","Type":"ContainerDied","Data":"4ad850f4634da0e2e217c2cb1a26088ad7d6bcab90e162d6eb724306d1b4f6bf"}
Oct 30 20:42:01.372914 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:01.372882    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x" event={"ID":"a2b7525a-f37d-48a2-bd4d-e763c8b155dc","Type":"ContainerStarted","Data":"ba49ae290f27189bc46a96b6551c86cf29c59783eb34c706b3cc96633abb7ca6"}
Oct 30 20:42:01.373281 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:01.372971    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:42:01.418805 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:01.418757    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x" podStartSLOduration=2.56961166 podStartE2EDuration="6.41874563s" totalImagesPullingTime="3.363071103s" totalInitContainerRuntime="486.062867ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:01.417249167 +0000 UTC m=+287.909065712" watchObservedRunningTime="2026-10-30 20:42:01.41874563 +0000 UTC m=+287.910562175"
Oct 30 20:42:12.380228 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:12.380197    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-pz44x"
Oct 30 20:42:13.684619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:13.684590    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:42:13.686594 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:13.686570    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:42:15.608952 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.608916    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"]
Oct 30 20:42:15.617035 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.617013    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"
Oct 30 20:42:15.625039 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.625014    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"]
Oct 30 20:42:15.626649 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.626628    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 20:42:15.626767 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.626667    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 20:42:15.631750 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.631732    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-22pnn\""
Oct 30 20:42:15.707075 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.707050    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be5c8e99-af95-453b-afb1-985a78e179ef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4dv9m\" (UID: \"be5c8e99-af95-453b-afb1-985a78e179ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"
Oct 30 20:42:15.707202 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.707130    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6fp6\" (UniqueName: \"kubernetes.io/projected/be5c8e99-af95-453b-afb1-985a78e179ef-kube-api-access-m6fp6\") pod \"cert-manager-webhook-5dbbbf75cc-4dv9m\" (UID: \"be5c8e99-af95-453b-afb1-985a78e179ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"
Oct 30 20:42:15.808031 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.807989    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6fp6\" (UniqueName: \"kubernetes.io/projected/be5c8e99-af95-453b-afb1-985a78e179ef-kube-api-access-m6fp6\") pod \"cert-manager-webhook-5dbbbf75cc-4dv9m\" (UID: \"be5c8e99-af95-453b-afb1-985a78e179ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"
Oct 30 20:42:15.808158 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.808090    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be5c8e99-af95-453b-afb1-985a78e179ef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4dv9m\" (UID: \"be5c8e99-af95-453b-afb1-985a78e179ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"
Oct 30 20:42:15.819525 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.819500    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6fp6\" (UniqueName: \"kubernetes.io/projected/be5c8e99-af95-453b-afb1-985a78e179ef-kube-api-access-m6fp6\") pod \"cert-manager-webhook-5dbbbf75cc-4dv9m\" (UID: \"be5c8e99-af95-453b-afb1-985a78e179ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"
Oct 30 20:42:15.828484 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.828455    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be5c8e99-af95-453b-afb1-985a78e179ef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4dv9m\" (UID: \"be5c8e99-af95-453b-afb1-985a78e179ef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"
Oct 30 20:42:15.934173 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:15.934147    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"
Oct 30 20:42:16.207009 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:16.206986    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"]
Oct 30 20:42:16.209068 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:42:16.209034    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe5c8e99_af95_453b_afb1_985a78e179ef.slice/crio-4ee0febf516579e7173bad1e5ce1a3b6da5556a55b84abe0c8d91a99c4a04673 WatchSource:0}: Error finding container 4ee0febf516579e7173bad1e5ce1a3b6da5556a55b84abe0c8d91a99c4a04673: Status 404 returned error can't find the container with id 4ee0febf516579e7173bad1e5ce1a3b6da5556a55b84abe0c8d91a99c4a04673
Oct 30 20:42:16.211009 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:16.210991    4810 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:42:16.438702 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:16.438673    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m" event={"ID":"be5c8e99-af95-453b-afb1-985a78e179ef","Type":"ContainerStarted","Data":"4ee0febf516579e7173bad1e5ce1a3b6da5556a55b84abe0c8d91a99c4a04673"}
Oct 30 20:42:20.459973 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:20.459934    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m" event={"ID":"be5c8e99-af95-453b-afb1-985a78e179ef","Type":"ContainerStarted","Data":"e20dd1af96ccf17bfa5509a31b7e459063fe7fa754ff1eed4eba1fc6dc915d86"}
Oct 30 20:42:20.460442 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:20.459994    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"
Oct 30 20:42:20.494566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:20.494526    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m" podStartSLOduration=2.185975243 podStartE2EDuration="5.494513552s" totalImagesPullingTime="3.308538309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:20.493959159 +0000 UTC m=+306.985775703" watchObservedRunningTime="2026-10-30 20:42:20.494513552 +0000 UTC m=+306.986330097"
Oct 30 20:42:21.283883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.283839    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"]
Oct 30 20:42:21.287608 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.287580    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.298059 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.298036    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 20:42:21.298192 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.298061    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 20:42:21.298192 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.298097    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 20:42:21.298333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.298120    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bdghc\""
Oct 30 20:42:21.298333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.298154    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 20:42:21.298333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.298282    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"]
Oct 30 20:42:21.362616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.362589    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6q7n\" (UniqueName: \"kubernetes.io/projected/b18bb0ae-131b-473b-b82e-e98ba3633842-kube-api-access-z6q7n\") pod \"kuadrant-console-plugin-66c4fb575d-cs5q5\" (UID: \"b18bb0ae-131b-473b-b82e-e98ba3633842\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.362735 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.362652    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b18bb0ae-131b-473b-b82e-e98ba3633842-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-cs5q5\" (UID: \"b18bb0ae-131b-473b-b82e-e98ba3633842\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.362792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.362745    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18bb0ae-131b-473b-b82e-e98ba3633842-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-cs5q5\" (UID: \"b18bb0ae-131b-473b-b82e-e98ba3633842\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.463115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.463094    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6q7n\" (UniqueName: \"kubernetes.io/projected/b18bb0ae-131b-473b-b82e-e98ba3633842-kube-api-access-z6q7n\") pod \"kuadrant-console-plugin-66c4fb575d-cs5q5\" (UID: \"b18bb0ae-131b-473b-b82e-e98ba3633842\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.463499 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.463172    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b18bb0ae-131b-473b-b82e-e98ba3633842-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-cs5q5\" (UID: \"b18bb0ae-131b-473b-b82e-e98ba3633842\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.463499 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.463203    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18bb0ae-131b-473b-b82e-e98ba3633842-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-cs5q5\" (UID: \"b18bb0ae-131b-473b-b82e-e98ba3633842\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.463828 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.463805    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b18bb0ae-131b-473b-b82e-e98ba3633842-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-cs5q5\" (UID: \"b18bb0ae-131b-473b-b82e-e98ba3633842\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.465684 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.465660    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18bb0ae-131b-473b-b82e-e98ba3633842-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-cs5q5\" (UID: \"b18bb0ae-131b-473b-b82e-e98ba3633842\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.474995 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.474976    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6q7n\" (UniqueName: \"kubernetes.io/projected/b18bb0ae-131b-473b-b82e-e98ba3633842-kube-api-access-z6q7n\") pod \"kuadrant-console-plugin-66c4fb575d-cs5q5\" (UID: \"b18bb0ae-131b-473b-b82e-e98ba3633842\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.599321 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.599265    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"
Oct 30 20:42:21.871667 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:21.871499    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5"]
Oct 30 20:42:21.874300 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:42:21.874270    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb18bb0ae_131b_473b_b82e_e98ba3633842.slice/crio-d7c57388a8dca8128097574f355a20d4be1016f3e451928f93b5802a32d1b9d2 WatchSource:0}: Error finding container d7c57388a8dca8128097574f355a20d4be1016f3e451928f93b5802a32d1b9d2: Status 404 returned error can't find the container with id d7c57388a8dca8128097574f355a20d4be1016f3e451928f93b5802a32d1b9d2
Oct 30 20:42:22.471071 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.471032    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5" event={"ID":"b18bb0ae-131b-473b-b82e-e98ba3633842","Type":"ContainerStarted","Data":"d7c57388a8dca8128097574f355a20d4be1016f3e451928f93b5802a32d1b9d2"}
Oct 30 20:42:22.777283 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.777207    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"]
Oct 30 20:42:22.782463 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.782442    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:22.795590 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.795566    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"]
Oct 30 20:42:22.849601 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.849579    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 20:42:22.849927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.849906    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 20:42:22.850040 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.850005    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6zrmq\""
Oct 30 20:42:22.874217 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.873853    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvv2m\" (UniqueName: \"kubernetes.io/projected/06a067b9-1e21-4355-b407-5ed910022940-kube-api-access-tvv2m\") pod \"kube-auth-proxy-64cbf78866-vhxkm\" (UID: \"06a067b9-1e21-4355-b407-5ed910022940\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:22.874217 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.874029    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06a067b9-1e21-4355-b407-5ed910022940-tmp\") pod \"kube-auth-proxy-64cbf78866-vhxkm\" (UID: \"06a067b9-1e21-4355-b407-5ed910022940\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:22.874217 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.874072    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a067b9-1e21-4355-b407-5ed910022940-tls-certs\") pod \"kube-auth-proxy-64cbf78866-vhxkm\" (UID: \"06a067b9-1e21-4355-b407-5ed910022940\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:22.974779 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.974745    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a067b9-1e21-4355-b407-5ed910022940-tls-certs\") pod \"kube-auth-proxy-64cbf78866-vhxkm\" (UID: \"06a067b9-1e21-4355-b407-5ed910022940\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:22.974983 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.974830    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvv2m\" (UniqueName: \"kubernetes.io/projected/06a067b9-1e21-4355-b407-5ed910022940-kube-api-access-tvv2m\") pod \"kube-auth-proxy-64cbf78866-vhxkm\" (UID: \"06a067b9-1e21-4355-b407-5ed910022940\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:22.975057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.975012    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06a067b9-1e21-4355-b407-5ed910022940-tmp\") pod \"kube-auth-proxy-64cbf78866-vhxkm\" (UID: \"06a067b9-1e21-4355-b407-5ed910022940\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:22.977692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.977666    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06a067b9-1e21-4355-b407-5ed910022940-tls-certs\") pod \"kube-auth-proxy-64cbf78866-vhxkm\" (UID: \"06a067b9-1e21-4355-b407-5ed910022940\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:22.977816 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.977796    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06a067b9-1e21-4355-b407-5ed910022940-tmp\") pod \"kube-auth-proxy-64cbf78866-vhxkm\" (UID: \"06a067b9-1e21-4355-b407-5ed910022940\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:22.989787 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:22.989748    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvv2m\" (UniqueName: \"kubernetes.io/projected/06a067b9-1e21-4355-b407-5ed910022940-kube-api-access-tvv2m\") pod \"kube-auth-proxy-64cbf78866-vhxkm\" (UID: \"06a067b9-1e21-4355-b407-5ed910022940\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:23.097694 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:23.097632    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"
Oct 30 20:42:23.381763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:23.381731    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm"]
Oct 30 20:42:23.386066 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:42:23.386039    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a067b9_1e21_4355_b407_5ed910022940.slice/crio-6a26e3fbabd952ffca39d4600af49313222c94dfa2f6d1b8f15c09a897db4947 WatchSource:0}: Error finding container 6a26e3fbabd952ffca39d4600af49313222c94dfa2f6d1b8f15c09a897db4947: Status 404 returned error can't find the container with id 6a26e3fbabd952ffca39d4600af49313222c94dfa2f6d1b8f15c09a897db4947
Oct 30 20:42:23.477367 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:23.477332    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm" event={"ID":"06a067b9-1e21-4355-b407-5ed910022940","Type":"ContainerStarted","Data":"6a26e3fbabd952ffca39d4600af49313222c94dfa2f6d1b8f15c09a897db4947"}
Oct 30 20:42:25.163073 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:42:25.162894    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:42:26.469698 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:26.468765    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4dv9m"
Oct 30 20:42:26.499904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:26.499572    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm" event={"ID":"06a067b9-1e21-4355-b407-5ed910022940","Type":"ContainerStarted","Data":"f8da57e879b3048e4bb88271c336e3a242ec7186e6052642fb81bdbfb46ccf9c"}
Oct 30 20:42:26.517587 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:26.517519    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64cbf78866-vhxkm" podStartSLOduration=2.1557955189999998 podStartE2EDuration="4.51750138s" totalImagesPullingTime="2.361705861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:26.516737998 +0000 UTC m=+313.008554545" watchObservedRunningTime="2026-10-30 20:42:26.51750138 +0000 UTC m=+313.009317927"
Oct 30 20:42:30.536766 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:30.536731    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5" event={"ID":"b18bb0ae-131b-473b-b82e-e98ba3633842","Type":"ContainerStarted","Data":"4fccae890eb9a37bc3b5ce53ad61e74a556c05feb73ef2d533bcf3c4500dd4f9"}
Oct 30 20:42:30.553373 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:30.553325    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cs5q5" podStartSLOduration=1.303924174 podStartE2EDuration="9.553308528s" totalImagesPullingTime="8.249384354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:30.552941301 +0000 UTC m=+317.044757847" watchObservedRunningTime="2026-10-30 20:42:30.553308528 +0000 UTC m=+317.045125076"
Oct 30 20:42:31.010788 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.010754    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"]
Oct 30 20:42:31.014744 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.014724    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"
Oct 30 20:42:31.024335 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.024310    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9gn9q\""
Oct 30 20:42:31.026279 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.026257    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"]
Oct 30 20:42:31.153425 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.153392    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7rkvg\" (UID: \"c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"
Oct 30 20:42:31.153570 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.153520    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6j7\" (UniqueName: \"kubernetes.io/projected/c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8-kube-api-access-8c6j7\") pod \"kuadrant-operator-controller-manager-678d87766f-7rkvg\" (UID: \"c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"
Oct 30 20:42:31.254638 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.254609    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6j7\" (UniqueName: \"kubernetes.io/projected/c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8-kube-api-access-8c6j7\") pod \"kuadrant-operator-controller-manager-678d87766f-7rkvg\" (UID: \"c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"
Oct 30 20:42:31.254767 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.254676    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7rkvg\" (UID: \"c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"
Oct 30 20:42:31.255035 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.255014    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7rkvg\" (UID: \"c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"
Oct 30 20:42:31.265082 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.265029    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6j7\" (UniqueName: \"kubernetes.io/projected/c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8-kube-api-access-8c6j7\") pod \"kuadrant-operator-controller-manager-678d87766f-7rkvg\" (UID: \"c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"
Oct 30 20:42:31.327879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.327834    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"
Oct 30 20:42:31.610259 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:31.610237    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"]
Oct 30 20:42:31.612324 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:42:31.612299    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc317aeb3_a1fe_44b4_bb5a_243dcb95a2e8.slice/crio-9d89b1f1c6d12049eca4b6fdd1de371831014bad8b32d1b61f3096d886753ec4 WatchSource:0}: Error finding container 9d89b1f1c6d12049eca4b6fdd1de371831014bad8b32d1b61f3096d886753ec4: Status 404 returned error can't find the container with id 9d89b1f1c6d12049eca4b6fdd1de371831014bad8b32d1b61f3096d886753ec4
Oct 30 20:42:32.552114 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:32.552066    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg" event={"ID":"c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8","Type":"ContainerStarted","Data":"9d89b1f1c6d12049eca4b6fdd1de371831014bad8b32d1b61f3096d886753ec4"}
Oct 30 20:42:36.574700 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:36.574658    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg" event={"ID":"c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8","Type":"ContainerStarted","Data":"f13098893874efd95f78f06a4c09cb5868d174b3901c3a1ddd21bdf4c5503cdf"}
Oct 30 20:42:36.575240 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:36.574767    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"
Oct 30 20:42:36.602132 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:36.602083    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg" podStartSLOduration=2.504148001 podStartE2EDuration="6.602070248s" totalImagesPullingTime="4.097922247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:36.600191969 +0000 UTC m=+323.092008515" watchObservedRunningTime="2026-10-30 20:42:36.602070248 +0000 UTC m=+323.093886793"
Oct 30 20:42:47.581588 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:42:47.581559    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7rkvg"
Oct 30 20:42:55.155101 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:42:55.155063    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:43:06.492269 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.492238    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:43:06.515245 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.515212    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:43:06.515428 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.515393    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"
Oct 30 20:43:06.517549 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.517526    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:43:06.519062 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.519044    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 20:43:06.551303 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.551279    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bcc30848-8e00-4f89-9f7b-a9f340c43866-config-file\") pod \"limitador-limitador-7c9b87d574-qwdhc\" (UID: \"bcc30848-8e00-4f89-9f7b-a9f340c43866\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"
Oct 30 20:43:06.551401 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.551344    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkbhw\" (UniqueName: \"kubernetes.io/projected/bcc30848-8e00-4f89-9f7b-a9f340c43866-kube-api-access-nkbhw\") pod \"limitador-limitador-7c9b87d574-qwdhc\" (UID: \"bcc30848-8e00-4f89-9f7b-a9f340c43866\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"
Oct 30 20:43:06.652255 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.652230    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bcc30848-8e00-4f89-9f7b-a9f340c43866-config-file\") pod \"limitador-limitador-7c9b87d574-qwdhc\" (UID: \"bcc30848-8e00-4f89-9f7b-a9f340c43866\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"
Oct 30 20:43:06.652353 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.652295    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkbhw\" (UniqueName: \"kubernetes.io/projected/bcc30848-8e00-4f89-9f7b-a9f340c43866-kube-api-access-nkbhw\") pod \"limitador-limitador-7c9b87d574-qwdhc\" (UID: \"bcc30848-8e00-4f89-9f7b-a9f340c43866\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"
Oct 30 20:43:06.652829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.652810    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bcc30848-8e00-4f89-9f7b-a9f340c43866-config-file\") pod \"limitador-limitador-7c9b87d574-qwdhc\" (UID: \"bcc30848-8e00-4f89-9f7b-a9f340c43866\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"
Oct 30 20:43:06.661691 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.661669    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkbhw\" (UniqueName: \"kubernetes.io/projected/bcc30848-8e00-4f89-9f7b-a9f340c43866-kube-api-access-nkbhw\") pod \"limitador-limitador-7c9b87d574-qwdhc\" (UID: \"bcc30848-8e00-4f89-9f7b-a9f340c43866\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"
Oct 30 20:43:06.829258 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:06.829195    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"
Oct 30 20:43:07.108009 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:07.107981    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:43:07.110338 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:43:07.110309    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc30848_8e00_4f89_9f7b_a9f340c43866.slice/crio-c4acfdb323eda438e7ee76169bfc2ee748dd6a841b6ead200c4191de603f6f11 WatchSource:0}: Error finding container c4acfdb323eda438e7ee76169bfc2ee748dd6a841b6ead200c4191de603f6f11: Status 404 returned error can't find the container with id c4acfdb323eda438e7ee76169bfc2ee748dd6a841b6ead200c4191de603f6f11
Oct 30 20:43:07.715810 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:07.715772    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc" event={"ID":"bcc30848-8e00-4f89-9f7b-a9f340c43866","Type":"ContainerStarted","Data":"c4acfdb323eda438e7ee76169bfc2ee748dd6a841b6ead200c4191de603f6f11"}
Oct 30 20:43:09.725409 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:09.725374    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc" event={"ID":"bcc30848-8e00-4f89-9f7b-a9f340c43866","Type":"ContainerStarted","Data":"2c637de25cccf01219dd2ac6413461d499bed022bacdcad45e5c005fd0626873"}
Oct 30 20:43:09.725795 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:09.725495    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"
Oct 30 20:43:09.745490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:09.745449    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc" podStartSLOduration=2.06491573 podStartE2EDuration="3.745435941s" totalImagesPullingTime="1.680520211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:09.743272267 +0000 UTC m=+356.235088811" watchObservedRunningTime="2026-10-30 20:43:09.745435941 +0000 UTC m=+356.237252485"
Oct 30 20:43:18.863065 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:18.863032    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"]
Oct 30 20:43:18.866090 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:18.866073    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:43:18.885743 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:18.885717    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"]
Oct 30 20:43:18.895670 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:18.895614    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 20:43:18.896503 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:18.896480    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cmmgz\""
Oct 30 20:43:18.934989 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:18.934966    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 20:43:18.935467 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:18.935444    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 20:43:18.973380 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:18.973350    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/426c9d87-cde0-47ad-a808-5547c7a1e885-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-jvp24\" (UID: \"426c9d87-cde0-47ad-a808-5547c7a1e885\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:43:18.973528 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:18.973398    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdkc2\" (UniqueName: \"kubernetes.io/projected/426c9d87-cde0-47ad-a808-5547c7a1e885-kube-api-access-xdkc2\") pod \"llmisvc-controller-manager-869d5dcc5d-jvp24\" (UID: \"426c9d87-cde0-47ad-a808-5547c7a1e885\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:43:19.074342 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:19.074314    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/426c9d87-cde0-47ad-a808-5547c7a1e885-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-jvp24\" (UID: \"426c9d87-cde0-47ad-a808-5547c7a1e885\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:43:19.074487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:19.074350    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkc2\" (UniqueName: \"kubernetes.io/projected/426c9d87-cde0-47ad-a808-5547c7a1e885-kube-api-access-xdkc2\") pod \"llmisvc-controller-manager-869d5dcc5d-jvp24\" (UID: \"426c9d87-cde0-47ad-a808-5547c7a1e885\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:43:19.076663 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:19.076642    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/426c9d87-cde0-47ad-a808-5547c7a1e885-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-jvp24\" (UID: \"426c9d87-cde0-47ad-a808-5547c7a1e885\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:43:19.089649 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:19.089626    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdkc2\" (UniqueName: \"kubernetes.io/projected/426c9d87-cde0-47ad-a808-5547c7a1e885-kube-api-access-xdkc2\") pod \"llmisvc-controller-manager-869d5dcc5d-jvp24\" (UID: \"426c9d87-cde0-47ad-a808-5547c7a1e885\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:43:19.186907 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:19.186884    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:43:19.465602 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:19.465531    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"]
Oct 30 20:43:19.468708 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:43:19.468681    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod426c9d87_cde0_47ad_a808_5547c7a1e885.slice/crio-0e85ff6cef4fe6fa1380c40702f2ba11a13635281f5d0864b6abec8ec087caa8 WatchSource:0}: Error finding container 0e85ff6cef4fe6fa1380c40702f2ba11a13635281f5d0864b6abec8ec087caa8: Status 404 returned error can't find the container with id 0e85ff6cef4fe6fa1380c40702f2ba11a13635281f5d0864b6abec8ec087caa8
Oct 30 20:43:19.774554 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:19.774469    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24" event={"ID":"426c9d87-cde0-47ad-a808-5547c7a1e885","Type":"ContainerStarted","Data":"0e85ff6cef4fe6fa1380c40702f2ba11a13635281f5d0864b6abec8ec087caa8"}
Oct 30 20:43:20.731629 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:20.731595    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"
Oct 30 20:43:21.786111 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:21.786075    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24" event={"ID":"426c9d87-cde0-47ad-a808-5547c7a1e885","Type":"ContainerStarted","Data":"f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535"}
Oct 30 20:43:21.786477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:21.786147    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:43:21.812728 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:21.812680    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24" podStartSLOduration=2.003252024 podStartE2EDuration="3.81266803s" totalImagesPullingTime="1.809416006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:21.81062057 +0000 UTC m=+368.302437116" watchObservedRunningTime="2026-10-30 20:43:21.81266803 +0000 UTC m=+368.304484574"
Oct 30 20:43:25.156287 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:43:25.156254    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:43:52.792851 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:43:52.792816    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:43:55.155292 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:43:55.155213    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:44:13.711394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:13.711365    4810 scope.go:122] "RemoveContainer" containerID="87f7a8d97c0a0c3a0fccf8c2d968bdfb22862bb8d4e92f7227baa0f9f58c4adb"
Oct 30 20:44:25.156014 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:44:25.155976    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:44:26.616961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:26.616925    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:44:37.292916 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.292884    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"]
Oct 30 20:44:37.296888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.296846    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.302291 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.302270    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:44:37.306314 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.306294    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 20:44:37.306446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.306294    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:44:37.306546 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.306339    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bbd9l\""
Oct 30 20:44:37.308460 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.308428    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"]
Oct 30 20:44:37.402916 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.402893    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:44:37.458964 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.458937    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx4bn\" (UniqueName: \"kubernetes.io/projected/9d56832d-9ea5-44de-a9cf-00d12d21f766-kube-api-access-jx4bn\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.459095 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.458995    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d56832d-9ea5-44de-a9cf-00d12d21f766-maas-api-tls\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.459160 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.459139    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d56832d-9ea5-44de-a9cf-00d12d21f766-metrics-certs\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.459197 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.459171    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d56832d-9ea5-44de-a9cf-00d12d21f766-openshift-service-ca\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.559996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.559933    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d56832d-9ea5-44de-a9cf-00d12d21f766-maas-api-tls\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.559996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.559982    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d56832d-9ea5-44de-a9cf-00d12d21f766-metrics-certs\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.560173 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.560001    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d56832d-9ea5-44de-a9cf-00d12d21f766-openshift-service-ca\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.560173 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.560048    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx4bn\" (UniqueName: \"kubernetes.io/projected/9d56832d-9ea5-44de-a9cf-00d12d21f766-kube-api-access-jx4bn\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.560723 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.560698    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d56832d-9ea5-44de-a9cf-00d12d21f766-openshift-service-ca\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.562381 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.562359    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d56832d-9ea5-44de-a9cf-00d12d21f766-metrics-certs\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.562483 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.562387    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d56832d-9ea5-44de-a9cf-00d12d21f766-maas-api-tls\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.569096 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.569071    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx4bn\" (UniqueName: \"kubernetes.io/projected/9d56832d-9ea5-44de-a9cf-00d12d21f766-kube-api-access-jx4bn\") pod \"maas-api-cbf9f78f-j5g8h\" (UID: \"9d56832d-9ea5-44de-a9cf-00d12d21f766\") " pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.610647 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.610625    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:37.886761 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:37.886724    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"]
Oct 30 20:44:37.888847 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:44:37.888821    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d56832d_9ea5_44de_a9cf_00d12d21f766.slice/crio-006a12cf898b45907c67251506917d63a3f750c3d9dd3a65eade565d0cbf6c61 WatchSource:0}: Error finding container 006a12cf898b45907c67251506917d63a3f750c3d9dd3a65eade565d0cbf6c61: Status 404 returned error can't find the container with id 006a12cf898b45907c67251506917d63a3f750c3d9dd3a65eade565d0cbf6c61
Oct 30 20:44:38.153996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:38.153923    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h" event={"ID":"9d56832d-9ea5-44de-a9cf-00d12d21f766","Type":"ContainerStarted","Data":"006a12cf898b45907c67251506917d63a3f750c3d9dd3a65eade565d0cbf6c61"}
Oct 30 20:44:40.435043 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:40.435009    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"]
Oct 30 20:44:40.438640 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:40.438606    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"
Oct 30 20:44:40.452115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:40.452093    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"]
Oct 30 20:44:40.484942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:40.484903    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fc88716-17fa-4b87-910f-a7ac94efc038-cert\") pod \"llmisvc-controller-manager-6466c554d8-qnpf4\" (UID: \"0fc88716-17fa-4b87-910f-a7ac94efc038\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"
Oct 30 20:44:40.485084 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:40.485020    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xtls\" (UniqueName: \"kubernetes.io/projected/0fc88716-17fa-4b87-910f-a7ac94efc038-kube-api-access-9xtls\") pod \"llmisvc-controller-manager-6466c554d8-qnpf4\" (UID: \"0fc88716-17fa-4b87-910f-a7ac94efc038\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"
Oct 30 20:44:40.586050 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:40.585981    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fc88716-17fa-4b87-910f-a7ac94efc038-cert\") pod \"llmisvc-controller-manager-6466c554d8-qnpf4\" (UID: \"0fc88716-17fa-4b87-910f-a7ac94efc038\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"
Oct 30 20:44:40.586050 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:40.586045    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtls\" (UniqueName: \"kubernetes.io/projected/0fc88716-17fa-4b87-910f-a7ac94efc038-kube-api-access-9xtls\") pod \"llmisvc-controller-manager-6466c554d8-qnpf4\" (UID: \"0fc88716-17fa-4b87-910f-a7ac94efc038\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"
Oct 30 20:44:40.588273 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:40.588253    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fc88716-17fa-4b87-910f-a7ac94efc038-cert\") pod \"llmisvc-controller-manager-6466c554d8-qnpf4\" (UID: \"0fc88716-17fa-4b87-910f-a7ac94efc038\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"
Oct 30 20:44:40.596496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:40.596471    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtls\" (UniqueName: \"kubernetes.io/projected/0fc88716-17fa-4b87-910f-a7ac94efc038-kube-api-access-9xtls\") pod \"llmisvc-controller-manager-6466c554d8-qnpf4\" (UID: \"0fc88716-17fa-4b87-910f-a7ac94efc038\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"
Oct 30 20:44:40.752280 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:40.752249    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"
Oct 30 20:44:41.025426 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:41.025401    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"]
Oct 30 20:44:41.027388 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:44:41.027356    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc88716_17fa_4b87_910f_a7ac94efc038.slice/crio-0819395388a9102cb6df773377d967ab485df8b6bb512b3808aa0f7aeffccbec WatchSource:0}: Error finding container 0819395388a9102cb6df773377d967ab485df8b6bb512b3808aa0f7aeffccbec: Status 404 returned error can't find the container with id 0819395388a9102cb6df773377d967ab485df8b6bb512b3808aa0f7aeffccbec
Oct 30 20:44:41.177962 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:41.177923    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h" event={"ID":"9d56832d-9ea5-44de-a9cf-00d12d21f766","Type":"ContainerStarted","Data":"1c9aa6c423bf566338c02c9cee1be2b5d257eb007b6842e8f02f2d7c39a4c8e1"}
Oct 30 20:44:41.178159 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:41.178055    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:41.179165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:41.179139    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4" event={"ID":"0fc88716-17fa-4b87-910f-a7ac94efc038","Type":"ContainerStarted","Data":"0819395388a9102cb6df773377d967ab485df8b6bb512b3808aa0f7aeffccbec"}
Oct 30 20:44:41.201900 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:41.201837    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h" podStartSLOduration=1.8266190839999998 podStartE2EDuration="4.201822559s" totalImagesPullingTime="2.375203475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:41.198120568 +0000 UTC m=+447.689937113" watchObservedRunningTime="2026-10-30 20:44:41.201822559 +0000 UTC m=+447.693639104"
Oct 30 20:44:42.186587 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:42.186552    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4" event={"ID":"0fc88716-17fa-4b87-910f-a7ac94efc038","Type":"ContainerStarted","Data":"8ccee773df092049f7feb496cced945899dd1c751516738c6d45c2b5f47b6052"}
Oct 30 20:44:42.187052 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:42.186626    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"
Oct 30 20:44:42.205821 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:42.205780    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4" podStartSLOduration=1.7481618970000001 podStartE2EDuration="2.205766833s" totalImagesPullingTime="457.604936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:42.205218943 +0000 UTC m=+448.697035489" watchObservedRunningTime="2026-10-30 20:44:42.205766833 +0000 UTC m=+448.697583355"
Oct 30 20:44:47.192030 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:44:47.191988    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cbf9f78f-j5g8h"
Oct 30 20:44:55.155617 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:44:55.155571    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:45:13.193184 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:13.193150    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qnpf4"
Oct 30 20:45:13.238325 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:13.238289    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"]
Oct 30 20:45:13.238540 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:13.238498    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24" podUID="426c9d87-cde0-47ad-a808-5547c7a1e885" containerName="manager" containerID="cri-o://f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535" gracePeriod=30
Oct 30 20:45:13.494225 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:13.494201    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:45:13.592113 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:13.592083    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426c9d87-cde0-47ad-a808-5547c7a1e885-cert\" (UniqueName: \"kubernetes.io/secret/426c9d87-cde0-47ad-a808-5547c7a1e885-cert\") pod \"426c9d87-cde0-47ad-a808-5547c7a1e885\" (UID: \"426c9d87-cde0-47ad-a808-5547c7a1e885\") "
Oct 30 20:45:13.592311 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:13.592215    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426c9d87-cde0-47ad-a808-5547c7a1e885-kube-api-access-xdkc2\" (UniqueName: \"kubernetes.io/projected/426c9d87-cde0-47ad-a808-5547c7a1e885-kube-api-access-xdkc2\") pod \"426c9d87-cde0-47ad-a808-5547c7a1e885\" (UID: \"426c9d87-cde0-47ad-a808-5547c7a1e885\") "
Oct 30 20:45:13.594267 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:13.594232    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426c9d87-cde0-47ad-a808-5547c7a1e885-cert" pod "426c9d87-cde0-47ad-a808-5547c7a1e885" (UID: "426c9d87-cde0-47ad-a808-5547c7a1e885"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:45:13.594388 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:13.594325    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426c9d87-cde0-47ad-a808-5547c7a1e885-kube-api-access-xdkc2" pod "426c9d87-cde0-47ad-a808-5547c7a1e885" (UID: "426c9d87-cde0-47ad-a808-5547c7a1e885"). InnerVolumeSpecName "kube-api-access-xdkc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:13.693740 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:13.693713    4810 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/426c9d87-cde0-47ad-a808-5547c7a1e885-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:45:13.693740 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:13.693739    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdkc2\" (UniqueName: \"kubernetes.io/projected/426c9d87-cde0-47ad-a808-5547c7a1e885-kube-api-access-xdkc2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:45:14.342602 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:14.342571    4810 generic.go:356] "Generic (PLEG): container finished" podID="426c9d87-cde0-47ad-a808-5547c7a1e885" containerID="f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535" exitCode=0
Oct 30 20:45:14.343041 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:14.342622    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"
Oct 30 20:45:14.343041 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:14.342624    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24" event={"ID":"426c9d87-cde0-47ad-a808-5547c7a1e885","Type":"ContainerDied","Data":"f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535"}
Oct 30 20:45:14.343041 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:14.342748    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24" event={"ID":"426c9d87-cde0-47ad-a808-5547c7a1e885","Type":"ContainerDied","Data":"0e85ff6cef4fe6fa1380c40702f2ba11a13635281f5d0864b6abec8ec087caa8"}
Oct 30 20:45:14.343041 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:14.342781    4810 scope.go:122] "RemoveContainer" containerID="f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535"
Oct 30 20:45:14.352142 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:14.352127    4810 scope.go:122] "RemoveContainer" containerID="f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535"
Oct 30 20:45:14.352433 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:45:14.352411    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535\": container with ID starting with f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535 not found: ID does not exist" containerID="f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535"
Oct 30 20:45:14.352498 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:14.352443    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535"} err="failed to get container status \"f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535\": rpc error: code = NotFound desc = could not find container \"f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535\": container with ID starting with f5c02a899799b8a8c4ce656ea9a7e66b40c9b8a75848b84643b156093115c535 not found: ID does not exist"
Oct 30 20:45:14.360103 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:14.360080    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"]
Oct 30 20:45:14.364977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:14.364958    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-jvp24"]
Oct 30 20:45:15.660459 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:15.660423    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426c9d87-cde0-47ad-a808-5547c7a1e885" path="/var/lib/kubelet/pods/426c9d87-cde0-47ad-a808-5547c7a1e885/volumes"
Oct 30 20:45:25.154551 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:45:25.154475    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:45:51.591563 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:51.591524    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:45:54.052051 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:54.052013    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:45:55.156216 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:45:55.156182    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:45:56.231666 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:45:56.231628    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:46:00.674672 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:00.674634    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:46:03.401015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:03.400976    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:46:06.990097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:06.990060    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"]
Oct 30 20:46:06.990698 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:06.990655    4810 cpu_manager.go:414] "containerMap: removing container" podUID="426c9d87-cde0-47ad-a808-5547c7a1e885" containerName="manager"
Oct 30 20:46:06.990698 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:06.990672    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426c9d87-cde0-47ad-a808-5547c7a1e885" containerName="manager"
Oct 30 20:46:06.990835 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:06.990773    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="426c9d87-cde0-47ad-a808-5547c7a1e885" containerName="manager"
Oct 30 20:46:06.994154 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:06.994132    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.005983 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.005966    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 20:46:07.010525 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.010500    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"]
Oct 30 20:46:07.051343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.051323    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h8j6c\""
Oct 30 20:46:07.093782 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.093752    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.093782 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.093784    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.093961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.093840    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.093961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.093889    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.093961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.093916    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.093961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.093935    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dssx\" (UniqueName: \"kubernetes.io/projected/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-kube-api-access-4dssx\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.094394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.094085    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.094394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.094202    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.094394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.094268    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.195507 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.195478    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.195656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.195520    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.195656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.195538    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.195656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.195562    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.195656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.195605    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.195656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.195624    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.195656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.195651    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dssx\" (UniqueName: \"kubernetes.io/projected/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-kube-api-access-4dssx\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.195991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.195684    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.195991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.195740    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.195991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.195972    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.196140 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.196044    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.196140 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.196060    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.196246 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.196175    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.196679 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.196648    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.198222 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.198196    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.198547 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.198526    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.204166 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.204145    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.204345 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.204322    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dssx\" (UniqueName: \"kubernetes.io/projected/f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30-kube-api-access-4dssx\") pod \"maas-default-gateway-openshift-default-77f556d647-d5dmn\" (UID: \"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.307191 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.307116    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:07.589108 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.589024    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"]
Oct 30 20:46:07.592451 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:46:07.592420    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ffe7b5_9ec3_4a68_acfc_df7dc69b5c30.slice/crio-d820cc0ca07b1a6412d42e8321d5acbb7933d07c6271a803e08aab825d90cd4e WatchSource:0}: Error finding container d820cc0ca07b1a6412d42e8321d5acbb7933d07c6271a803e08aab825d90cd4e: Status 404 returned error can't find the container with id d820cc0ca07b1a6412d42e8321d5acbb7933d07c6271a803e08aab825d90cd4e
Oct 30 20:46:07.612137 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:07.612101    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" event={"ID":"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30","Type":"ContainerStarted","Data":"d820cc0ca07b1a6412d42e8321d5acbb7933d07c6271a803e08aab825d90cd4e"}
Oct 30 20:46:08.176080 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:08.176041    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:46:11.169328 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:11.169298    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:46:11.169633 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:11.169362    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:46:11.169633 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:11.169389    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:46:11.640402 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:11.640366    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" event={"ID":"f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30","Type":"ContainerStarted","Data":"e7fc2dc2ddecd310b1daaba429a9b0800b29d9d963e0e2e821386abacd2ab7ca"}
Oct 30 20:46:11.663742 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:11.663691    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" podStartSLOduration=2.09005796 podStartE2EDuration="5.663677663s" totalImagesPullingTime="3.573619703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:11.662024584 +0000 UTC m=+538.153841131" watchObservedRunningTime="2026-10-30 20:46:11.663677663 +0000 UTC m=+538.155494273"
Oct 30 20:46:12.308196 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:12.308159    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:12.309658 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:12.309631    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d5dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 20:46:12.309774 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:12.309706    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" podUID="f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 20:46:13.307977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:13.307943    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d5dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 20:46:13.308154 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:13.308010    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" podUID="f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 20:46:14.308167 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:14.308122    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d5dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 20:46:14.308622 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:14.308180    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" podUID="f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 20:46:15.308374 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:15.308333    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d5dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 20:46:15.308897 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:15.308409    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" podUID="f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 20:46:16.307465 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:16.307428    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d5dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 20:46:16.307674 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:16.307485    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" podUID="f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 20:46:17.308146 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:17.308103    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d5dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 20:46:17.308570 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:17.308177    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" podUID="f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 20:46:18.307471 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:18.307436    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d5dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 20:46:18.307643 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:18.307497    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" podUID="f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 20:46:19.353333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:19.353293    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:19.354623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:19.354597    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:19.355207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:46:19.355183    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn"
Oct 30 20:46:25.157773 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:46:25.157730    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:46:55.155982 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:46:55.155840    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:47:00.638586 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:00.638546    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:47:06.850939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:06.850906    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:47:11.624311 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.624280    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"]
Oct 30 20:47:11.628042 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.628024    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:11.634328 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.634306    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:47:11.634530 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.634400    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:47:11.635015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.634992    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"]
Oct 30 20:47:11.730983 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.730949    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bps2\" (UniqueName: \"kubernetes.io/projected/d7e70fad-16c5-40d5-b40d-40320ada2363-kube-api-access-9bps2\") pod \"maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:11.731147 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.731020    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:11.731212 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.731189    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:11.832039 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.832002    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:11.832239 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.832087    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:11.832239 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.832110    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bps2\" (UniqueName: \"kubernetes.io/projected/d7e70fad-16c5-40d5-b40d-40320ada2363-kube-api-access-9bps2\") pod \"maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:11.832870 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.832828    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:11.834548 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.834513    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:11.844230 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.844204    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bps2\" (UniqueName: \"kubernetes.io/projected/d7e70fad-16c5-40d5-b40d-40320ada2363-kube-api-access-9bps2\") pod \"maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:11.943686 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:11.943650    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:12.225388 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:12.225359    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"]
Oct 30 20:47:12.227411 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:47:12.227383    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e70fad_16c5_40d5_b40d_40320ada2363.slice/crio-d6c51f1ee90d8d8b108a013093447c8d7637a6538a78fe18ff24fc28f1f1e1c2 WatchSource:0}: Error finding container d6c51f1ee90d8d8b108a013093447c8d7637a6538a78fe18ff24fc28f1f1e1c2: Status 404 returned error can't find the container with id d6c51f1ee90d8d8b108a013093447c8d7637a6538a78fe18ff24fc28f1f1e1c2
Oct 30 20:47:12.957517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:12.957475    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6" event={"ID":"d7e70fad-16c5-40d5-b40d-40320ada2363","Type":"ContainerStarted","Data":"d6c51f1ee90d8d8b108a013093447c8d7637a6538a78fe18ff24fc28f1f1e1c2"}
Oct 30 20:47:13.721513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:13.721486    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:47:13.726611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:13.726583    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:47:13.943101 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:13.943066    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:47:15.977209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:15.977108    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6" event={"ID":"d7e70fad-16c5-40d5-b40d-40320ada2363","Type":"ContainerStarted","Data":"712050a65911405e27a2ec8cabb48eb6752bd146d681013ceceda23f57d711ca"}
Oct 30 20:47:15.999013 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:15.998955    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6" podStartSLOduration=1.8672919220000002 podStartE2EDuration="4.998941951s" totalImagesPullingTime="3.131650029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:15.997494001 +0000 UTC m=+602.489310548" watchObservedRunningTime="2026-10-30 20:47:15.998941951 +0000 UTC m=+602.490758495"
Oct 30 20:47:16.917708 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:16.917674    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"]
Oct 30 20:47:16.921542 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:16.921518    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:16.929471 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:16.929439    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"]
Oct 30 20:47:16.982652 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:16.982624    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8c9s\" (UniqueName: \"kubernetes.io/projected/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-kube-api-access-z8c9s\") pod \"maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:16.983095 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:16.982666    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:16.983095 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:16.982783    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:16.983853 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:16.983829    4810 generic.go:356] "Generic (PLEG): container finished" podID="d7e70fad-16c5-40d5-b40d-40320ada2363" containerID="712050a65911405e27a2ec8cabb48eb6752bd146d681013ceceda23f57d711ca" exitCode=0
Oct 30 20:47:16.983984 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:16.983898    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6" event={"ID":"d7e70fad-16c5-40d5-b40d-40320ada2363","Type":"ContainerDied","Data":"712050a65911405e27a2ec8cabb48eb6752bd146d681013ceceda23f57d711ca"}
Oct 30 20:47:17.083579 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:17.083547    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:17.083749 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:17.083677    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8c9s\" (UniqueName: \"kubernetes.io/projected/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-kube-api-access-z8c9s\") pod \"maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:17.083749 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:17.083716    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:17.084357 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:17.084334    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:17.085954 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:17.085929    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:17.093318 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:17.093289    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8c9s\" (UniqueName: \"kubernetes.io/projected/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-kube-api-access-z8c9s\") pod \"maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:17.240044 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:17.239967    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:17.517589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:17.517509    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"]
Oct 30 20:47:17.521046 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:47:17.521020    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d171e95_a676_421e_b6ec_2c0f74fc9ba2.slice/crio-39b2b71003ce8e2da89eb954bda26f056f72e32c2c24c348ed2ff8096da1ef84 WatchSource:0}: Error finding container 39b2b71003ce8e2da89eb954bda26f056f72e32c2c24c348ed2ff8096da1ef84: Status 404 returned error can't find the container with id 39b2b71003ce8e2da89eb954bda26f056f72e32c2c24c348ed2ff8096da1ef84
Oct 30 20:47:17.991784 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:17.991745    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2" event={"ID":"6d171e95-a676-421e-b6ec-2c0f74fc9ba2","Type":"ContainerStarted","Data":"20035cd676bedfba88509b9cb5f10fca15aada8732e15c60e0c600aa867ef1d3"}
Oct 30 20:47:17.992239 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:17.991807    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2" event={"ID":"6d171e95-a676-421e-b6ec-2c0f74fc9ba2","Type":"ContainerStarted","Data":"39b2b71003ce8e2da89eb954bda26f056f72e32c2c24c348ed2ff8096da1ef84"}
Oct 30 20:47:18.015698 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.015641    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2" podStartSLOduration=2.015624809 podStartE2EDuration="2.015624809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:18.01341443 +0000 UTC m=+604.505230976" watchObservedRunningTime="2026-10-30 20:47:18.015624809 +0000 UTC m=+604.507441331"
Oct 30 20:47:18.127460 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.127438    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:18.194225 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.194196    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-openshift-ca\") pod \"d7e70fad-16c5-40d5-b40d-40320ada2363\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") "
Oct 30 20:47:18.194426 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.194324    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e70fad-16c5-40d5-b40d-40320ada2363-kube-api-access-9bps2\" (UniqueName: \"kubernetes.io/projected/d7e70fad-16c5-40d5-b40d-40320ada2363-kube-api-access-9bps2\") pod \"d7e70fad-16c5-40d5-b40d-40320ada2363\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") "
Oct 30 20:47:18.194426 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.194377    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-xks-ca\") pod \"d7e70fad-16c5-40d5-b40d-40320ada2363\" (UID: \"d7e70fad-16c5-40d5-b40d-40320ada2363\") "
Oct 30 20:47:18.194576 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.194531    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-openshift-ca" pod "d7e70fad-16c5-40d5-b40d-40320ada2363" (UID: "d7e70fad-16c5-40d5-b40d-40320ada2363"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:18.194701 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.194686    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.196545 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.196518    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-xks-ca" pod "d7e70fad-16c5-40d5-b40d-40320ada2363" (UID: "d7e70fad-16c5-40d5-b40d-40320ada2363"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:18.196632 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.196580    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e70fad-16c5-40d5-b40d-40320ada2363-kube-api-access-9bps2" pod "d7e70fad-16c5-40d5-b40d-40320ada2363" (UID: "d7e70fad-16c5-40d5-b40d-40320ada2363"). InnerVolumeSpecName "kube-api-access-9bps2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:18.295299 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.295246    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e70fad-16c5-40d5-b40d-40320ada2363-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.295299 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.295275    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bps2\" (UniqueName: \"kubernetes.io/projected/d7e70fad-16c5-40d5-b40d-40320ada2363-kube-api-access-9bps2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:18.997445 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.997413    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"
Oct 30 20:47:18.997445 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.997423    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6" event={"ID":"d7e70fad-16c5-40d5-b40d-40320ada2363","Type":"ContainerDied","Data":"d6c51f1ee90d8d8b108a013093447c8d7637a6538a78fe18ff24fc28f1f1e1c2"}
Oct 30 20:47:18.997954 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.997478    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6c51f1ee90d8d8b108a013093447c8d7637a6538a78fe18ff24fc28f1f1e1c2"
Oct 30 20:47:18.999237 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.999213    4810 generic.go:356] "Generic (PLEG): container finished" podID="6d171e95-a676-421e-b6ec-2c0f74fc9ba2" containerID="20035cd676bedfba88509b9cb5f10fca15aada8732e15c60e0c600aa867ef1d3" exitCode=0
Oct 30 20:47:18.999363 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:18.999296    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2" event={"ID":"6d171e95-a676-421e-b6ec-2c0f74fc9ba2","Type":"ContainerDied","Data":"20035cd676bedfba88509b9cb5f10fca15aada8732e15c60e0c600aa867ef1d3"}
Oct 30 20:47:20.145352 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:20.145326    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:20.216535 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:20.216506    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-openshift-ca\") pod \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") "
Oct 30 20:47:20.216701 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:20.216594    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-kube-api-access-z8c9s\" (UniqueName: \"kubernetes.io/projected/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-kube-api-access-z8c9s\") pod \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") "
Oct 30 20:47:20.216701 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:20.216634    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-xks-ca\") pod \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\" (UID: \"6d171e95-a676-421e-b6ec-2c0f74fc9ba2\") "
Oct 30 20:47:20.216882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:20.216834    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-openshift-ca" pod "6d171e95-a676-421e-b6ec-2c0f74fc9ba2" (UID: "6d171e95-a676-421e-b6ec-2c0f74fc9ba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:20.218676 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:20.218655    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-kube-api-access-z8c9s" pod "6d171e95-a676-421e-b6ec-2c0f74fc9ba2" (UID: "6d171e95-a676-421e-b6ec-2c0f74fc9ba2"). InnerVolumeSpecName "kube-api-access-z8c9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:20.218781 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:20.218724    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-xks-ca" pod "6d171e95-a676-421e-b6ec-2c0f74fc9ba2" (UID: "6d171e95-a676-421e-b6ec-2c0f74fc9ba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:20.318344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:20.318262    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:20.318344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:20.318293    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8c9s\" (UniqueName: \"kubernetes.io/projected/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-kube-api-access-z8c9s\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:20.318344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:20.318305    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d171e95-a676-421e-b6ec-2c0f74fc9ba2-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:21.011026 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:21.010996    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"
Oct 30 20:47:21.011026 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:21.011005    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2" event={"ID":"6d171e95-a676-421e-b6ec-2c0f74fc9ba2","Type":"ContainerDied","Data":"39b2b71003ce8e2da89eb954bda26f056f72e32c2c24c348ed2ff8096da1ef84"}
Oct 30 20:47:21.011235 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:21.011057    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39b2b71003ce8e2da89eb954bda26f056f72e32c2c24c348ed2ff8096da1ef84"
Oct 30 20:47:23.772520 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.772484    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"]
Oct 30 20:47:23.773070 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.773052    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6d171e95-a676-421e-b6ec-2c0f74fc9ba2" containerName="revoke-keys"
Oct 30 20:47:23.773114 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.773072    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d171e95-a676-421e-b6ec-2c0f74fc9ba2" containerName="revoke-keys"
Oct 30 20:47:23.773114 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.773094    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d7e70fad-16c5-40d5-b40d-40320ada2363" containerName="revoke-keys"
Oct 30 20:47:23.773114 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.773100    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e70fad-16c5-40d5-b40d-40320ada2363" containerName="revoke-keys"
Oct 30 20:47:23.773210 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.773188    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d171e95-a676-421e-b6ec-2c0f74fc9ba2" containerName="revoke-keys"
Oct 30 20:47:23.773210 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.773194    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e70fad-16c5-40d5-b40d-40320ada2363" containerName="revoke-keys"
Oct 30 20:47:23.776474 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.776458    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:23.781408 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.781387    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:47:23.781537 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.781452    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:47:23.783284 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.783262    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"]
Oct 30 20:47:23.853944 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.853913    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b16c21be-b679g\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:23.854092 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.854024    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8nlk\" (UniqueName: \"kubernetes.io/projected/a62781f9-5fab-46b3-91ea-5e2765deafe6-kube-api-access-n8nlk\") pod \"maas-api-revoke-keys-e2e-ait-b16c21be-b679g\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:23.854092 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.854070    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b16c21be-b679g\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:23.955088 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.955057    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8nlk\" (UniqueName: \"kubernetes.io/projected/a62781f9-5fab-46b3-91ea-5e2765deafe6-kube-api-access-n8nlk\") pod \"maas-api-revoke-keys-e2e-ait-b16c21be-b679g\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:23.955253 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.955092    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b16c21be-b679g\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:23.955253 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.955179    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b16c21be-b679g\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:23.955843 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.955823    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b16c21be-b679g\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:23.957563 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.957542    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b16c21be-b679g\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:23.963573 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:23.963537    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8nlk\" (UniqueName: \"kubernetes.io/projected/a62781f9-5fab-46b3-91ea-5e2765deafe6-kube-api-access-n8nlk\") pod \"maas-api-revoke-keys-e2e-ait-b16c21be-b679g\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:24.092898 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:24.092801    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:24.364190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:24.364166    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"]
Oct 30 20:47:24.366238 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:47:24.366216    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62781f9_5fab_46b3_91ea_5e2765deafe6.slice/crio-529a9de061f93c8e7c42a84fcf25035c1e8a14e9d580647df839c40356e07f04 WatchSource:0}: Error finding container 529a9de061f93c8e7c42a84fcf25035c1e8a14e9d580647df839c40356e07f04: Status 404 returned error can't find the container with id 529a9de061f93c8e7c42a84fcf25035c1e8a14e9d580647df839c40356e07f04
Oct 30 20:47:25.036413 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.036338    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g" event={"ID":"a62781f9-5fab-46b3-91ea-5e2765deafe6","Type":"ContainerStarted","Data":"786788361e6aa883b38ef8ea44ea5bda490150a5e2e61b567750008b57a620fb"}
Oct 30 20:47:25.036413 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.036387    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g" event={"ID":"a62781f9-5fab-46b3-91ea-5e2765deafe6","Type":"ContainerStarted","Data":"529a9de061f93c8e7c42a84fcf25035c1e8a14e9d580647df839c40356e07f04"}
Oct 30 20:47:25.053153 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.053107    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g" podStartSLOduration=2.053093796 podStartE2EDuration="2.053093796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:25.052899025 +0000 UTC m=+611.544715570" watchObservedRunningTime="2026-10-30 20:47:25.053093796 +0000 UTC m=+611.544910338"
Oct 30 20:47:25.155406 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:47:25.155374    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:47:25.674730 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.674691    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"]
Oct 30 20:47:25.678096 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.678075    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:25.705967 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.705936    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"]
Oct 30 20:47:25.773123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.773087    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:25.773123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.773126    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpbsf\" (UniqueName: \"kubernetes.io/projected/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-kube-api-access-xpbsf\") pod \"maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:25.773314 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.773152    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:25.874536 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.874489    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:25.874703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.874543    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpbsf\" (UniqueName: \"kubernetes.io/projected/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-kube-api-access-xpbsf\") pod \"maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:25.874703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.874582    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:25.875545 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.875512    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:25.876977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.876953    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:25.884790 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.884761    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpbsf\" (UniqueName: \"kubernetes.io/projected/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-kube-api-access-xpbsf\") pod \"maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:25.991409 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:25.991328    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:26.050025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:26.044938    4810 generic.go:356] "Generic (PLEG): container finished" podID="a62781f9-5fab-46b3-91ea-5e2765deafe6" containerID="786788361e6aa883b38ef8ea44ea5bda490150a5e2e61b567750008b57a620fb" exitCode=0
Oct 30 20:47:26.050025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:26.045002    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g" event={"ID":"a62781f9-5fab-46b3-91ea-5e2765deafe6","Type":"ContainerDied","Data":"786788361e6aa883b38ef8ea44ea5bda490150a5e2e61b567750008b57a620fb"}
Oct 30 20:47:26.250158 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:26.250074    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"]
Oct 30 20:47:26.253603 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:26.253580    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b612c7f2-s2gg6"]
Oct 30 20:47:26.278785 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:26.278760    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"]
Oct 30 20:47:26.281528 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:47:26.281503    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac64ab0_c2bc_46de_bb16_72a7fcb61d66.slice/crio-5fafcf84c7ee13375f5f1d63a6e0cf4fa0e959554a8400380a825fb889bdfe1b WatchSource:0}: Error finding container 5fafcf84c7ee13375f5f1d63a6e0cf4fa0e959554a8400380a825fb889bdfe1b: Status 404 returned error can't find the container with id 5fafcf84c7ee13375f5f1d63a6e0cf4fa0e959554a8400380a825fb889bdfe1b
Oct 30 20:47:27.054982 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.054926    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk" event={"ID":"aac64ab0-c2bc-46de-bb16-72a7fcb61d66","Type":"ContainerStarted","Data":"a5a4bb126b158538a56fba12ce33614219f4bf4d78798ad37c1976eb7dec0989"}
Oct 30 20:47:27.055498 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.054999    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk" event={"ID":"aac64ab0-c2bc-46de-bb16-72a7fcb61d66","Type":"ContainerStarted","Data":"5fafcf84c7ee13375f5f1d63a6e0cf4fa0e959554a8400380a825fb889bdfe1b"}
Oct 30 20:47:27.076826 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.075936    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk" podStartSLOduration=2.0759159719999998 podStartE2EDuration="2.075915972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:27.072292183 +0000 UTC m=+613.564108730" watchObservedRunningTime="2026-10-30 20:47:27.075915972 +0000 UTC m=+613.567732517"
Oct 30 20:47:27.265568 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.265541    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:27.390518 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.390489    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-openshift-ca\") pod \"a62781f9-5fab-46b3-91ea-5e2765deafe6\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") "
Oct 30 20:47:27.390688 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.390567    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-xks-ca\") pod \"a62781f9-5fab-46b3-91ea-5e2765deafe6\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") "
Oct 30 20:47:27.390688 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.390596    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62781f9-5fab-46b3-91ea-5e2765deafe6-kube-api-access-n8nlk\" (UniqueName: \"kubernetes.io/projected/a62781f9-5fab-46b3-91ea-5e2765deafe6-kube-api-access-n8nlk\") pod \"a62781f9-5fab-46b3-91ea-5e2765deafe6\" (UID: \"a62781f9-5fab-46b3-91ea-5e2765deafe6\") "
Oct 30 20:47:27.390845 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.390822    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-openshift-ca" pod "a62781f9-5fab-46b3-91ea-5e2765deafe6" (UID: "a62781f9-5fab-46b3-91ea-5e2765deafe6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:27.390992 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.390973    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:27.392722 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.392693    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-xks-ca" pod "a62781f9-5fab-46b3-91ea-5e2765deafe6" (UID: "a62781f9-5fab-46b3-91ea-5e2765deafe6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:27.392722 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.392711    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62781f9-5fab-46b3-91ea-5e2765deafe6-kube-api-access-n8nlk" pod "a62781f9-5fab-46b3-91ea-5e2765deafe6" (UID: "a62781f9-5fab-46b3-91ea-5e2765deafe6"). InnerVolumeSpecName "kube-api-access-n8nlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:27.491717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.491680    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62781f9-5fab-46b3-91ea-5e2765deafe6-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:27.491717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.491713    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8nlk\" (UniqueName: \"kubernetes.io/projected/a62781f9-5fab-46b3-91ea-5e2765deafe6-kube-api-access-n8nlk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:27.660783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:27.660708    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e70fad-16c5-40d5-b40d-40320ada2363" path="/var/lib/kubelet/pods/d7e70fad-16c5-40d5-b40d-40320ada2363/volumes"
Oct 30 20:47:28.060825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:28.060740    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"
Oct 30 20:47:28.060825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:28.060762    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g" event={"ID":"a62781f9-5fab-46b3-91ea-5e2765deafe6","Type":"ContainerDied","Data":"529a9de061f93c8e7c42a84fcf25035c1e8a14e9d580647df839c40356e07f04"}
Oct 30 20:47:28.061257 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:28.060828    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="529a9de061f93c8e7c42a84fcf25035c1e8a14e9d580647df839c40356e07f04"
Oct 30 20:47:28.062355 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:28.062331    4810 generic.go:356] "Generic (PLEG): container finished" podID="aac64ab0-c2bc-46de-bb16-72a7fcb61d66" containerID="a5a4bb126b158538a56fba12ce33614219f4bf4d78798ad37c1976eb7dec0989" exitCode=0
Oct 30 20:47:28.062468 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:28.062406    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk" event={"ID":"aac64ab0-c2bc-46de-bb16-72a7fcb61d66","Type":"ContainerDied","Data":"a5a4bb126b158538a56fba12ce33614219f4bf4d78798ad37c1976eb7dec0989"}
Oct 30 20:47:28.230541 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:28.230497    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:47:29.266773 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.266746    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:29.413738 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.413707    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-xks-ca\") pod \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") "
Oct 30 20:47:29.413921 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.413748    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-openshift-ca\") pod \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") "
Oct 30 20:47:29.413921 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.413793    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-kube-api-access-xpbsf\" (UniqueName: \"kubernetes.io/projected/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-kube-api-access-xpbsf\") pod \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\" (UID: \"aac64ab0-c2bc-46de-bb16-72a7fcb61d66\") "
Oct 30 20:47:29.414238 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.414213    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-openshift-ca" pod "aac64ab0-c2bc-46de-bb16-72a7fcb61d66" (UID: "aac64ab0-c2bc-46de-bb16-72a7fcb61d66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:29.416054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.416018    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-xks-ca" pod "aac64ab0-c2bc-46de-bb16-72a7fcb61d66" (UID: "aac64ab0-c2bc-46de-bb16-72a7fcb61d66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:29.416154 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.416099    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-kube-api-access-xpbsf" pod "aac64ab0-c2bc-46de-bb16-72a7fcb61d66" (UID: "aac64ab0-c2bc-46de-bb16-72a7fcb61d66"). InnerVolumeSpecName "kube-api-access-xpbsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:29.514940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.514907    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:29.514940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.514932    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:29.514940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.514942    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpbsf\" (UniqueName: \"kubernetes.io/projected/aac64ab0-c2bc-46de-bb16-72a7fcb61d66-kube-api-access-xpbsf\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:47:29.944580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.944499    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"]
Oct 30 20:47:29.950091 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:29.950065    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f46c771-v2qc2"]
Oct 30 20:47:30.076315 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:30.076281    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"
Oct 30 20:47:30.076315 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:30.076296    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk" event={"ID":"aac64ab0-c2bc-46de-bb16-72a7fcb61d66","Type":"ContainerDied","Data":"5fafcf84c7ee13375f5f1d63a6e0cf4fa0e959554a8400380a825fb889bdfe1b"}
Oct 30 20:47:30.076510 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:30.076351    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fafcf84c7ee13375f5f1d63a6e0cf4fa0e959554a8400380a825fb889bdfe1b"
Oct 30 20:47:31.661184 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:31.661144    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d171e95-a676-421e-b6ec-2c0f74fc9ba2" path="/var/lib/kubelet/pods/6d171e95-a676-421e-b6ec-2c0f74fc9ba2/volumes"
Oct 30 20:47:34.386317 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:34.386280    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:47:37.328229 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:37.327022    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"]
Oct 30 20:47:37.333614 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:37.333583    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eaf151e7-hhkkk"]
Oct 30 20:47:37.662821 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:37.662786    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac64ab0-c2bc-46de-bb16-72a7fcb61d66" path="/var/lib/kubelet/pods/aac64ab0-c2bc-46de-bb16-72a7fcb61d66/volumes"
Oct 30 20:47:38.796962 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:38.796929    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"]
Oct 30 20:47:38.801216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:38.801188    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16c21be-b679g"]
Oct 30 20:47:39.660994 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:39.660960    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62781f9-5fab-46b3-91ea-5e2765deafe6" path="/var/lib/kubelet/pods/a62781f9-5fab-46b3-91ea-5e2765deafe6/volumes"
Oct 30 20:47:41.956040 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:41.956003    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:47:48.194060 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:48.194018    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:47:55.156414 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:47:55.156370    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:47:55.944163 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:47:55.944132    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:48:03.970224 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:03.970185    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:48:04.751550 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.751515    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"]
Oct 30 20:48:04.752013 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.751998    4810 cpu_manager.go:414] "containerMap: removing container" podUID="aac64ab0-c2bc-46de-bb16-72a7fcb61d66" containerName="revoke-keys"
Oct 30 20:48:04.752066 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.752040    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac64ab0-c2bc-46de-bb16-72a7fcb61d66" containerName="revoke-keys"
Oct 30 20:48:04.752066 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.752061    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a62781f9-5fab-46b3-91ea-5e2765deafe6" containerName="revoke-keys"
Oct 30 20:48:04.752139 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.752067    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62781f9-5fab-46b3-91ea-5e2765deafe6" containerName="revoke-keys"
Oct 30 20:48:04.752191 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.752180    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac64ab0-c2bc-46de-bb16-72a7fcb61d66" containerName="revoke-keys"
Oct 30 20:48:04.752231 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.752195    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62781f9-5fab-46b3-91ea-5e2765deafe6" containerName="revoke-keys"
Oct 30 20:48:04.755426 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.755409    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:04.766985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.766956    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"]
Oct 30 20:48:04.767188 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.767067    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-317a33\"/\"kube-root-ca.crt\""
Oct 30 20:48:04.767318 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.767086    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-317a33\"/\"openshift-service-ca.crt\""
Oct 30 20:48:04.767318 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.767111    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-317a33\"/\"default-dockercfg-pwtr5\""
Oct 30 20:48:04.767916 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.767896    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-317a33\"/\"simulator-w1-317a33-kserve-self-signed-certs\""
Oct 30 20:48:04.900926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.900894    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-model-cache\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:04.901096 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.900941    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-dshm\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:04.901179 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.901102    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk5nh\" (UniqueName: \"kubernetes.io/projected/ab0e213f-d8cf-4659-829f-d7629ace07d4-kube-api-access-fk5nh\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:04.901248 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.901225    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0e213f-d8cf-4659-829f-d7629ace07d4-tls-certs\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:04.901304 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.901287    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-home\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:04.901354 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:04.901318    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-tmp-dir\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.003087 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.002994    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk5nh\" (UniqueName: \"kubernetes.io/projected/ab0e213f-d8cf-4659-829f-d7629ace07d4-kube-api-access-fk5nh\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.003549 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.003094    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0e213f-d8cf-4659-829f-d7629ace07d4-tls-certs\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.003549 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.003141    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-home\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.003549 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.003165    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-tmp-dir\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.003549 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.003207    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-model-cache\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.003549 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.003246    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-dshm\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.003829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.003655    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-tmp-dir\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.003829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.003685    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-model-cache\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.003975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.003896    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-home\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.006136 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.006088    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-dshm\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.006272 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.006175    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0e213f-d8cf-4659-829f-d7629ace07d4-tls-certs\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.018970 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.018902    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk5nh\" (UniqueName: \"kubernetes.io/projected/ab0e213f-d8cf-4659-829f-d7629ace07d4-kube-api-access-fk5nh\") pod \"simulator-w1-317a33-kserve-8c7446bf8-mdzwb\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") " pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.073676 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.073643    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:05.372307 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.372223    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"]
Oct 30 20:48:05.375528 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:48:05.375502    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab0e213f_d8cf_4659_829f_d7629ace07d4.slice/crio-59b4e34064240aef7bdb2c4da2c57988147db31c0850b5fcf6ea7f6ee2b7bde8 WatchSource:0}: Error finding container 59b4e34064240aef7bdb2c4da2c57988147db31c0850b5fcf6ea7f6ee2b7bde8: Status 404 returned error can't find the container with id 59b4e34064240aef7bdb2c4da2c57988147db31c0850b5fcf6ea7f6ee2b7bde8
Oct 30 20:48:05.377599 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:05.377579    4810 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:48:06.287189 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:06.287153    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" event={"ID":"ab0e213f-d8cf-4659-829f-d7629ace07d4","Type":"ContainerStarted","Data":"59b4e34064240aef7bdb2c4da2c57988147db31c0850b5fcf6ea7f6ee2b7bde8"}
Oct 30 20:48:07.296293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:07.296257    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" event={"ID":"ab0e213f-d8cf-4659-829f-d7629ace07d4","Type":"ContainerStarted","Data":"cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74"}
Oct 30 20:48:07.327265 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:07.327214    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podStartSLOduration=2.065188641 podStartE2EDuration="3.327196133s" totalImagesPullingTime="1.262007492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:07.325344058 +0000 UTC m=+653.817160602" watchObservedRunningTime="2026-10-30 20:48:07.327196133 +0000 UTC m=+653.819012678"
Oct 30 20:48:08.074015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:08.073985    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:11.866414 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:11.866378    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:48:15.074078 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:15.074039    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:15.088266 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:15.088240    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:15.355000 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:15.354920    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:48:21.309735 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:21.309694    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:48:25.157184 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:48:25.157107    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:48:30.241617 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.241582    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"]
Oct 30 20:48:30.245465 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.245443    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.251876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.251811    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-53aa47\"/\"premium-w4-53aa47-kserve-self-signed-certs\""
Oct 30 20:48:30.251876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.251822    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-53aa47\"/\"default-dockercfg-b57dx\""
Oct 30 20:48:30.252176 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.252153    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-53aa47\"/\"openshift-service-ca.crt\""
Oct 30 20:48:30.252379 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.252363    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-53aa47\"/\"kube-root-ca.crt\""
Oct 30 20:48:30.261162 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.261132    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"]
Oct 30 20:48:30.337548 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.337505    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-dshm\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.337548 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.337545    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-model-cache\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.337788 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.337657    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tls-certs\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.337850 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.337791    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-home\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.337850 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.337824    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tmp-dir\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.337990 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.337951    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdv9m\" (UniqueName: \"kubernetes.io/projected/71c1e45e-3f48-47e8-b216-77d2a7d6afca-kube-api-access-rdv9m\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.438420 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.438390    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-home\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.438420 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.438424    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tmp-dir\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.438657 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.438470    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdv9m\" (UniqueName: \"kubernetes.io/projected/71c1e45e-3f48-47e8-b216-77d2a7d6afca-kube-api-access-rdv9m\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.438657 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.438500    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-dshm\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.438657 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.438522    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-model-cache\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.438657 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.438569    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tls-certs\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.438876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.438816    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-home\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.438939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.438896    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tmp-dir\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.439003 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.438980    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-model-cache\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.441011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.440987    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-dshm\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.441352 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.441330    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tls-certs\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.451212 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.451185    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdv9m\" (UniqueName: \"kubernetes.io/projected/71c1e45e-3f48-47e8-b216-77d2a7d6afca-kube-api-access-rdv9m\") pod \"premium-w4-53aa47-kserve-7f99bc6c95-gmg8w\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") " pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.558177 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.558100    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:30.848125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:30.848099    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"]
Oct 30 20:48:30.851397 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:48:30.851371    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71c1e45e_3f48_47e8_b216_77d2a7d6afca.slice/crio-d48389cf48b9f6889de73b94ab33714c6849fb4fa4fa355b12503a96c32766ba WatchSource:0}: Error finding container d48389cf48b9f6889de73b94ab33714c6849fb4fa4fa355b12503a96c32766ba: Status 404 returned error can't find the container with id d48389cf48b9f6889de73b94ab33714c6849fb4fa4fa355b12503a96c32766ba
Oct 30 20:48:31.436722 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:31.436680    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" event={"ID":"71c1e45e-3f48-47e8-b216-77d2a7d6afca","Type":"ContainerStarted","Data":"d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac"}
Oct 30 20:48:31.437227 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:31.436737    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" event={"ID":"71c1e45e-3f48-47e8-b216-77d2a7d6afca","Type":"ContainerStarted","Data":"d48389cf48b9f6889de73b94ab33714c6849fb4fa4fa355b12503a96c32766ba"}
Oct 30 20:48:31.459061 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:31.459006    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podStartSLOduration=1.4589937769999999 podStartE2EDuration="1.458993777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:31.457061293 +0000 UTC m=+677.948877836" watchObservedRunningTime="2026-10-30 20:48:31.458993777 +0000 UTC m=+677.950810323"
Oct 30 20:48:31.558953 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:31.558923    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:33.889352 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:33.889311    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:48:36.423284 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.423256    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"]
Oct 30 20:48:36.428025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.427999    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.445236 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.445208    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bc102e\"/\"openshift-service-ca.crt\""
Oct 30 20:48:36.445371 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.445351    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bc102e\"/\"premium-w0-bc102e-kserve-self-signed-certs\""
Oct 30 20:48:36.446767 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.446746    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"]
Oct 30 20:48:36.449704 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.449686    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bc102e\"/\"kube-root-ca.crt\""
Oct 30 20:48:36.450362 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.450342    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bc102e\"/\"default-dockercfg-s2c6d\""
Oct 30 20:48:36.502532 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.502491    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-dshm\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.502678 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.502549    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vslm\" (UniqueName: \"kubernetes.io/projected/9abdc7e8-e62f-455f-94b4-603fb02a88f8-kube-api-access-6vslm\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.502731 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.502706    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tmp-dir\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.502770 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.502761    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-home\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.502953 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.502929    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-model-cache\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.503058 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.503043    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tls-certs\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.603578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.603540    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tls-certs\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.603578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.603577    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-dshm\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.603831 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.603597    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vslm\" (UniqueName: \"kubernetes.io/projected/9abdc7e8-e62f-455f-94b4-603fb02a88f8-kube-api-access-6vslm\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.603831 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.603626    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tmp-dir\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.603831 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.603663    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-home\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.603831 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.603741    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-model-cache\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.604174 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.604142    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tmp-dir\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.604290 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.604178    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-home\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.604290 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.604209    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-model-cache\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.606330 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.606306    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-dshm\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.606636 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.606620    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tls-certs\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.616585 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.616503    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vslm\" (UniqueName: \"kubernetes.io/projected/9abdc7e8-e62f-455f-94b4-603fb02a88f8-kube-api-access-6vslm\") pod \"premium-w0-bc102e-kserve-7cdc978d6c-c6png\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") " pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:36.741584 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:36.741508    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:37.036819 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:37.036790    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"]
Oct 30 20:48:37.039438 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:48:37.039396    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9abdc7e8_e62f_455f_94b4_603fb02a88f8.slice/crio-913a37ac5af59d5d18e739f43493e1e046c1bc4c3d28fb42d8541b4ca0e948c4 WatchSource:0}: Error finding container 913a37ac5af59d5d18e739f43493e1e046c1bc4c3d28fb42d8541b4ca0e948c4: Status 404 returned error can't find the container with id 913a37ac5af59d5d18e739f43493e1e046c1bc4c3d28fb42d8541b4ca0e948c4
Oct 30 20:48:37.474446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:37.474403    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" event={"ID":"9abdc7e8-e62f-455f-94b4-603fb02a88f8","Type":"ContainerStarted","Data":"4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321"}
Oct 30 20:48:37.474446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:37.474459    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" event={"ID":"9abdc7e8-e62f-455f-94b4-603fb02a88f8","Type":"ContainerStarted","Data":"913a37ac5af59d5d18e739f43493e1e046c1bc4c3d28fb42d8541b4ca0e948c4"}
Oct 30 20:48:37.498394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:37.498345    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podStartSLOduration=1.498331994 podStartE2EDuration="1.498331994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:37.495823703 +0000 UTC m=+683.987640248" watchObservedRunningTime="2026-10-30 20:48:37.498331994 +0000 UTC m=+683.990148538"
Oct 30 20:48:37.741653 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:37.741583    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:40.558445 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:40.558409    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:40.570838 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:40.570813    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:41.511678 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:41.511602    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:48:46.707485 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:46.707450    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:48:46.742575 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:46.742541    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:46.755179 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:46.755155    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:47.546404 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:47.546372    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:48:55.154780 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:48:55.154741    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:48:59.225335 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.225301    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"]
Oct 30 20:48:59.228974 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.228954    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.235848 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.235823    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:48:59.236037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.236020    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:48:59.240415 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.240390    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"]
Oct 30 20:48:59.241547 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.241527    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27950da6-tkzft\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.241656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.241585    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27950da6-tkzft\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.241725 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.241709    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jshp\" (UniqueName: \"kubernetes.io/projected/d69ce9d3-933d-4fc7-922a-b9755d5d2285-kube-api-access-2jshp\") pod \"maas-api-revoke-sub-27950da6-tkzft\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.342799 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.342761    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27950da6-tkzft\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.342975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.342816    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jshp\" (UniqueName: \"kubernetes.io/projected/d69ce9d3-933d-4fc7-922a-b9755d5d2285-kube-api-access-2jshp\") pod \"maas-api-revoke-sub-27950da6-tkzft\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.342975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.342916    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27950da6-tkzft\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.343442 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.343420    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27950da6-tkzft\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.345210 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.345189    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27950da6-tkzft\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.358215 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.358194    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jshp\" (UniqueName: \"kubernetes.io/projected/d69ce9d3-933d-4fc7-922a-b9755d5d2285-kube-api-access-2jshp\") pod \"maas-api-revoke-sub-27950da6-tkzft\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.543837 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.543761    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:48:59.820619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:48:59.820588    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"]
Oct 30 20:48:59.822212 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:48:59.822170    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69ce9d3_933d_4fc7_922a_b9755d5d2285.slice/crio-a6bec84453fe00bde2bf72f1c0f18163ee78b597df1a64536a68123746997bab WatchSource:0}: Error finding container a6bec84453fe00bde2bf72f1c0f18163ee78b597df1a64536a68123746997bab: Status 404 returned error can't find the container with id a6bec84453fe00bde2bf72f1c0f18163ee78b597df1a64536a68123746997bab
Oct 30 20:49:00.608403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:00.608369    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft" event={"ID":"d69ce9d3-933d-4fc7-922a-b9755d5d2285","Type":"ContainerStarted","Data":"1bfb3bc7fd966d74b5dabfbe93f2ea75270dca94d9bfd3bbb215b10a966a7b8d"}
Oct 30 20:49:00.608781 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:00.608423    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft" event={"ID":"d69ce9d3-933d-4fc7-922a-b9755d5d2285","Type":"ContainerStarted","Data":"a6bec84453fe00bde2bf72f1c0f18163ee78b597df1a64536a68123746997bab"}
Oct 30 20:49:00.627933 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:00.627882    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft" podStartSLOduration=1.627869397 podStartE2EDuration="1.627869397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:00.62620021 +0000 UTC m=+707.118016755" watchObservedRunningTime="2026-10-30 20:49:00.627869397 +0000 UTC m=+707.119685934"
Oct 30 20:49:01.616557 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:01.616513    4810 generic.go:356] "Generic (PLEG): container finished" podID="d69ce9d3-933d-4fc7-922a-b9755d5d2285" containerID="1bfb3bc7fd966d74b5dabfbe93f2ea75270dca94d9bfd3bbb215b10a966a7b8d" exitCode=0
Oct 30 20:49:01.617099 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:01.616567    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft" event={"ID":"d69ce9d3-933d-4fc7-922a-b9755d5d2285","Type":"ContainerDied","Data":"1bfb3bc7fd966d74b5dabfbe93f2ea75270dca94d9bfd3bbb215b10a966a7b8d"}
Oct 30 20:49:02.773977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:02.773943    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:49:02.877786 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:02.877762    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-openshift-ca\") pod \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") "
Oct 30 20:49:02.877958 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:02.877836    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69ce9d3-933d-4fc7-922a-b9755d5d2285-kube-api-access-2jshp\" (UniqueName: \"kubernetes.io/projected/d69ce9d3-933d-4fc7-922a-b9755d5d2285-kube-api-access-2jshp\") pod \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") "
Oct 30 20:49:02.878024 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:02.877982    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-xks-ca\") pod \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\" (UID: \"d69ce9d3-933d-4fc7-922a-b9755d5d2285\") "
Oct 30 20:49:02.878209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:02.878184    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-openshift-ca" pod "d69ce9d3-933d-4fc7-922a-b9755d5d2285" (UID: "d69ce9d3-933d-4fc7-922a-b9755d5d2285"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:02.878336 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:02.878321    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:02.880303 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:02.880271    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69ce9d3-933d-4fc7-922a-b9755d5d2285-kube-api-access-2jshp" pod "d69ce9d3-933d-4fc7-922a-b9755d5d2285" (UID: "d69ce9d3-933d-4fc7-922a-b9755d5d2285"). InnerVolumeSpecName "kube-api-access-2jshp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:02.880456 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:02.880441    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-xks-ca" pod "d69ce9d3-933d-4fc7-922a-b9755d5d2285" (UID: "d69ce9d3-933d-4fc7-922a-b9755d5d2285"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:02.979165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:02.979137    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69ce9d3-933d-4fc7-922a-b9755d5d2285-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:02.979165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:02.979163    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jshp\" (UniqueName: \"kubernetes.io/projected/d69ce9d3-933d-4fc7-922a-b9755d5d2285-kube-api-access-2jshp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:03.629971 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:03.629946    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"
Oct 30 20:49:03.630147 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:03.629943    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft" event={"ID":"d69ce9d3-933d-4fc7-922a-b9755d5d2285","Type":"ContainerDied","Data":"a6bec84453fe00bde2bf72f1c0f18163ee78b597df1a64536a68123746997bab"}
Oct 30 20:49:03.630147 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:03.630063    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6bec84453fe00bde2bf72f1c0f18163ee78b597df1a64536a68123746997bab"
Oct 30 20:49:04.293704 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:04.293659    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:49:10.700219 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.700185    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"]
Oct 30 20:49:10.700892 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.700871    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d69ce9d3-933d-4fc7-922a-b9755d5d2285" containerName="revoke-keys"
Oct 30 20:49:10.700991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.700892    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69ce9d3-933d-4fc7-922a-b9755d5d2285" containerName="revoke-keys"
Oct 30 20:49:10.701049 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.701038    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69ce9d3-933d-4fc7-922a-b9755d5d2285" containerName="revoke-keys"
Oct 30 20:49:10.705498 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.705477    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:10.710684 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.710661    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:49:10.710795 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.710731    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:49:10.714046 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.714024    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"]
Oct 30 20:49:10.756394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.756368    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgplr\" (UniqueName: \"kubernetes.io/projected/df9ad61e-d3ae-4607-a169-ee97d925699f-kube-api-access-pgplr\") pod \"maas-api-revoke-sub-b0080b10-2gdsl\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:10.756515 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.756495    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0080b10-2gdsl\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:10.756569 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.756553    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0080b10-2gdsl\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:10.857347 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.857317    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgplr\" (UniqueName: \"kubernetes.io/projected/df9ad61e-d3ae-4607-a169-ee97d925699f-kube-api-access-pgplr\") pod \"maas-api-revoke-sub-b0080b10-2gdsl\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:10.857479 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.857388    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0080b10-2gdsl\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:10.857479 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.857413    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0080b10-2gdsl\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:10.858085 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.858060    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0080b10-2gdsl\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:10.859841 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.859814    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0080b10-2gdsl\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:10.871054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:10.871033    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgplr\" (UniqueName: \"kubernetes.io/projected/df9ad61e-d3ae-4607-a169-ee97d925699f-kube-api-access-pgplr\") pod \"maas-api-revoke-sub-b0080b10-2gdsl\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:11.024075 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.023999    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:11.304034 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.304010    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"]
Oct 30 20:49:11.305321 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:49:11.305291    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9ad61e_d3ae_4607_a169_ee97d925699f.slice/crio-715f660668598af5fc3e0803d9d280c9ce97aff7c78522e9eb9e0fcec3a4dd99 WatchSource:0}: Error finding container 715f660668598af5fc3e0803d9d280c9ce97aff7c78522e9eb9e0fcec3a4dd99: Status 404 returned error can't find the container with id 715f660668598af5fc3e0803d9d280c9ce97aff7c78522e9eb9e0fcec3a4dd99
Oct 30 20:49:11.676179 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.676146    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl" event={"ID":"df9ad61e-d3ae-4607-a169-ee97d925699f","Type":"ContainerStarted","Data":"8ff57fa8db13ca2949dc1d4d27ba072ca6826ae28d1ed3103ed615cd25668544"}
Oct 30 20:49:11.676338 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.676200    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl" event={"ID":"df9ad61e-d3ae-4607-a169-ee97d925699f","Type":"ContainerStarted","Data":"715f660668598af5fc3e0803d9d280c9ce97aff7c78522e9eb9e0fcec3a4dd99"}
Oct 30 20:49:11.696189 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.696139    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl" podStartSLOduration=1.696127078 podStartE2EDuration="1.696127078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:11.694267276 +0000 UTC m=+718.186083821" watchObservedRunningTime="2026-10-30 20:49:11.696127078 +0000 UTC m=+718.187943622"
Oct 30 20:49:11.850393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.850313    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"]
Oct 30 20:49:11.854893 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.854871    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:11.865405 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.865369    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"]
Oct 30 20:49:11.968912 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.968875    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0752c29f-55bx8\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:11.969063 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.968959    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0752c29f-55bx8\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:11.969063 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:11.969017    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rktps\" (UniqueName: \"kubernetes.io/projected/19b25634-57cc-4352-af9b-17e10f50978c-kube-api-access-rktps\") pod \"maas-api-revoke-sub-0752c29f-55bx8\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:12.069951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.069912    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0752c29f-55bx8\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:12.070133 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.069977    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0752c29f-55bx8\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:12.070133 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.070002    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rktps\" (UniqueName: \"kubernetes.io/projected/19b25634-57cc-4352-af9b-17e10f50978c-kube-api-access-rktps\") pod \"maas-api-revoke-sub-0752c29f-55bx8\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:12.070666 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.070638    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0752c29f-55bx8\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:12.072269 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.072245    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0752c29f-55bx8\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:12.080148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.080127    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rktps\" (UniqueName: \"kubernetes.io/projected/19b25634-57cc-4352-af9b-17e10f50978c-kube-api-access-rktps\") pod \"maas-api-revoke-sub-0752c29f-55bx8\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:12.189938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.189910    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:12.468538 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.468510    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"]
Oct 30 20:49:12.470609 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:49:12.470581    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b25634_57cc_4352_af9b_17e10f50978c.slice/crio-3c89b0838f02574628007476b6d92263656eb046694aaff50661f28a679c7734 WatchSource:0}: Error finding container 3c89b0838f02574628007476b6d92263656eb046694aaff50661f28a679c7734: Status 404 returned error can't find the container with id 3c89b0838f02574628007476b6d92263656eb046694aaff50661f28a679c7734
Oct 30 20:49:12.682918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.682885    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8" event={"ID":"19b25634-57cc-4352-af9b-17e10f50978c","Type":"ContainerStarted","Data":"3c89b0838f02574628007476b6d92263656eb046694aaff50661f28a679c7734"}
Oct 30 20:49:12.684504 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.684476    4810 generic.go:356] "Generic (PLEG): container finished" podID="df9ad61e-d3ae-4607-a169-ee97d925699f" containerID="8ff57fa8db13ca2949dc1d4d27ba072ca6826ae28d1ed3103ed615cd25668544" exitCode=0
Oct 30 20:49:12.684628 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:12.684559    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl" event={"ID":"df9ad61e-d3ae-4607-a169-ee97d925699f","Type":"ContainerDied","Data":"8ff57fa8db13ca2949dc1d4d27ba072ca6826ae28d1ed3103ed615cd25668544"}
Oct 30 20:49:13.691530 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.691501    4810 generic.go:356] "Generic (PLEG): container finished" podID="19b25634-57cc-4352-af9b-17e10f50978c" containerID="8760370e4f4cf0f8e0af85ef7af8651d86a8108d05ec36dd1d44c6401c8b0bb1" exitCode=0
Oct 30 20:49:13.691954 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.691528    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8" event={"ID":"19b25634-57cc-4352-af9b-17e10f50978c","Type":"ContainerDied","Data":"8760370e4f4cf0f8e0af85ef7af8651d86a8108d05ec36dd1d44c6401c8b0bb1"}
Oct 30 20:49:13.835370 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.835348    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:13.887255 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.887224    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9ad61e-d3ae-4607-a169-ee97d925699f-kube-api-access-pgplr\" (UniqueName: \"kubernetes.io/projected/df9ad61e-d3ae-4607-a169-ee97d925699f-kube-api-access-pgplr\") pod \"df9ad61e-d3ae-4607-a169-ee97d925699f\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") "
Oct 30 20:49:13.887412 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.887263    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-xks-ca\") pod \"df9ad61e-d3ae-4607-a169-ee97d925699f\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") "
Oct 30 20:49:13.887412 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.887315    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-openshift-ca\") pod \"df9ad61e-d3ae-4607-a169-ee97d925699f\" (UID: \"df9ad61e-d3ae-4607-a169-ee97d925699f\") "
Oct 30 20:49:13.887724 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.887681    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-openshift-ca" pod "df9ad61e-d3ae-4607-a169-ee97d925699f" (UID: "df9ad61e-d3ae-4607-a169-ee97d925699f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:13.889228 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.889205    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9ad61e-d3ae-4607-a169-ee97d925699f-kube-api-access-pgplr" pod "df9ad61e-d3ae-4607-a169-ee97d925699f" (UID: "df9ad61e-d3ae-4607-a169-ee97d925699f"). InnerVolumeSpecName "kube-api-access-pgplr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:13.889363 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.889344    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-xks-ca" pod "df9ad61e-d3ae-4607-a169-ee97d925699f" (UID: "df9ad61e-d3ae-4607-a169-ee97d925699f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:13.988984 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.988911    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgplr\" (UniqueName: \"kubernetes.io/projected/df9ad61e-d3ae-4607-a169-ee97d925699f-kube-api-access-pgplr\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:13.988984 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.988943    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:13.988984 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:13.988958    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9ad61e-d3ae-4607-a169-ee97d925699f-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:14.699177 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.699140    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl" event={"ID":"df9ad61e-d3ae-4607-a169-ee97d925699f","Type":"ContainerDied","Data":"715f660668598af5fc3e0803d9d280c9ce97aff7c78522e9eb9e0fcec3a4dd99"}
Oct 30 20:49:14.699562 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.699207    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="715f660668598af5fc3e0803d9d280c9ce97aff7c78522e9eb9e0fcec3a4dd99"
Oct 30 20:49:14.699562 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.699155    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"
Oct 30 20:49:14.844079 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.844056    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:14.896937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.896903    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b25634-57cc-4352-af9b-17e10f50978c-kube-api-access-rktps\" (UniqueName: \"kubernetes.io/projected/19b25634-57cc-4352-af9b-17e10f50978c-kube-api-access-rktps\") pod \"19b25634-57cc-4352-af9b-17e10f50978c\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") "
Oct 30 20:49:14.897155 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.897130    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-xks-ca\") pod \"19b25634-57cc-4352-af9b-17e10f50978c\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") "
Oct 30 20:49:14.897262 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.897247    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-openshift-ca\") pod \"19b25634-57cc-4352-af9b-17e10f50978c\" (UID: \"19b25634-57cc-4352-af9b-17e10f50978c\") "
Oct 30 20:49:14.897655 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.897623    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-openshift-ca" pod "19b25634-57cc-4352-af9b-17e10f50978c" (UID: "19b25634-57cc-4352-af9b-17e10f50978c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:14.899210 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.899185    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-xks-ca" pod "19b25634-57cc-4352-af9b-17e10f50978c" (UID: "19b25634-57cc-4352-af9b-17e10f50978c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:14.899560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.899531    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b25634-57cc-4352-af9b-17e10f50978c-kube-api-access-rktps" pod "19b25634-57cc-4352-af9b-17e10f50978c" (UID: "19b25634-57cc-4352-af9b-17e10f50978c"). InnerVolumeSpecName "kube-api-access-rktps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:14.998332 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.998264    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:14.998332 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.998292    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b25634-57cc-4352-af9b-17e10f50978c-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:14.998332 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:14.998303    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rktps\" (UniqueName: \"kubernetes.io/projected/19b25634-57cc-4352-af9b-17e10f50978c-kube-api-access-rktps\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:15.706780 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:15.706753    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"
Oct 30 20:49:15.707149 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:15.706781    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8" event={"ID":"19b25634-57cc-4352-af9b-17e10f50978c","Type":"ContainerDied","Data":"3c89b0838f02574628007476b6d92263656eb046694aaff50661f28a679c7734"}
Oct 30 20:49:15.707149 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:15.706840    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c89b0838f02574628007476b6d92263656eb046694aaff50661f28a679c7734"
Oct 30 20:49:24.343332 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.343294    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"]
Oct 30 20:49:24.343746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.343732    4810 cpu_manager.go:414] "containerMap: removing container" podUID="19b25634-57cc-4352-af9b-17e10f50978c" containerName="revoke-keys"
Oct 30 20:49:24.343791 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.343747    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b25634-57cc-4352-af9b-17e10f50978c" containerName="revoke-keys"
Oct 30 20:49:24.343791 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.343778    4810 cpu_manager.go:414] "containerMap: removing container" podUID="df9ad61e-d3ae-4607-a169-ee97d925699f" containerName="revoke-keys"
Oct 30 20:49:24.343791 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.343782    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9ad61e-d3ae-4607-a169-ee97d925699f" containerName="revoke-keys"
Oct 30 20:49:24.343905 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.343884    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9ad61e-d3ae-4607-a169-ee97d925699f" containerName="revoke-keys"
Oct 30 20:49:24.343905 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.343895    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b25634-57cc-4352-af9b-17e10f50978c" containerName="revoke-keys"
Oct 30 20:49:24.347704 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.347687    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.354907 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.353754    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:49:24.354907 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.353791    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:49:24.357428 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.356737    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"]
Oct 30 20:49:24.493105 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.493066    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68a3758-5h5jj\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.493272 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.493126    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68a3758-5h5jj\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.493319 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.493275    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrwnm\" (UniqueName: \"kubernetes.io/projected/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-kube-api-access-mrwnm\") pod \"maas-api-revoke-sub-a68a3758-5h5jj\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.593887 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.593795    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwnm\" (UniqueName: \"kubernetes.io/projected/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-kube-api-access-mrwnm\") pod \"maas-api-revoke-sub-a68a3758-5h5jj\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.594025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.593890    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68a3758-5h5jj\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.594025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.593917    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68a3758-5h5jj\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.594500 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.594477    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68a3758-5h5jj\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.596299 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.596270    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68a3758-5h5jj\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.604011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.603787    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrwnm\" (UniqueName: \"kubernetes.io/projected/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-kube-api-access-mrwnm\") pod \"maas-api-revoke-sub-a68a3758-5h5jj\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.664729 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.664702    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:24.945054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.945003    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"]
Oct 30 20:49:24.949009 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:49:24.948829    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6aa8b9c_9151_4b39_af65_541dbba0ccd7.slice/crio-df966a96d6ab2a4b203e6cb8148968de3efe8b13b8666f1b90d8eb0c271dedee WatchSource:0}: Error finding container df966a96d6ab2a4b203e6cb8148968de3efe8b13b8666f1b90d8eb0c271dedee: Status 404 returned error can't find the container with id df966a96d6ab2a4b203e6cb8148968de3efe8b13b8666f1b90d8eb0c271dedee
Oct 30 20:49:24.958611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.958576    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"]
Oct 30 20:49:24.964108 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.964080    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:24.972697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:24.972673    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"]
Oct 30 20:49:25.099287 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.099250    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4adaa4a7-2hk8z\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:25.099469 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.099372    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q927r\" (UniqueName: \"kubernetes.io/projected/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-kube-api-access-q927r\") pod \"maas-api-revoke-sub-4adaa4a7-2hk8z\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:25.099469 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.099444    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4adaa4a7-2hk8z\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:25.155729 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:49:25.155692    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:49:25.200792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.200731    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q927r\" (UniqueName: \"kubernetes.io/projected/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-kube-api-access-q927r\") pod \"maas-api-revoke-sub-4adaa4a7-2hk8z\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:25.200792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.200764    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4adaa4a7-2hk8z\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:25.200986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.200902    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4adaa4a7-2hk8z\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:25.201704 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.201674    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4adaa4a7-2hk8z\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:25.203131 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.203098    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4adaa4a7-2hk8z\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:25.210218 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.210195    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q927r\" (UniqueName: \"kubernetes.io/projected/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-kube-api-access-q927r\") pod \"maas-api-revoke-sub-4adaa4a7-2hk8z\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:25.290717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.290694    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:25.577729 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.577699    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"]
Oct 30 20:49:25.578518 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:49:25.578496    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b0e915_e59e_4f23_b4e4_c68bcb4e48e5.slice/crio-785bc65b9693ba7a257e8240fefec5f67b3312f7309903f0bedcf2d6989fbffd WatchSource:0}: Error finding container 785bc65b9693ba7a257e8240fefec5f67b3312f7309903f0bedcf2d6989fbffd: Status 404 returned error can't find the container with id 785bc65b9693ba7a257e8240fefec5f67b3312f7309903f0bedcf2d6989fbffd
Oct 30 20:49:25.764198 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.764160    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj" event={"ID":"f6aa8b9c-9151-4b39-af65-541dbba0ccd7","Type":"ContainerStarted","Data":"26f1428769fe2082ca66f31e286b681de722ac61d8c8ee2f7c037667e8fa447c"}
Oct 30 20:49:25.764391 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.764229    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj" event={"ID":"f6aa8b9c-9151-4b39-af65-541dbba0ccd7","Type":"ContainerStarted","Data":"df966a96d6ab2a4b203e6cb8148968de3efe8b13b8666f1b90d8eb0c271dedee"}
Oct 30 20:49:25.765609 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.765582    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z" event={"ID":"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5","Type":"ContainerStarted","Data":"785bc65b9693ba7a257e8240fefec5f67b3312f7309903f0bedcf2d6989fbffd"}
Oct 30 20:49:25.783392 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:25.783339    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj" podStartSLOduration=1.783321127 podStartE2EDuration="1.783321127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:25.78110784 +0000 UTC m=+732.272924388" watchObservedRunningTime="2026-10-30 20:49:25.783321127 +0000 UTC m=+732.275137675"
Oct 30 20:49:26.774297 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:26.774202    4810 generic.go:356] "Generic (PLEG): container finished" podID="f6aa8b9c-9151-4b39-af65-541dbba0ccd7" containerID="26f1428769fe2082ca66f31e286b681de722ac61d8c8ee2f7c037667e8fa447c" exitCode=0
Oct 30 20:49:26.774703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:26.774286    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj" event={"ID":"f6aa8b9c-9151-4b39-af65-541dbba0ccd7","Type":"ContainerDied","Data":"26f1428769fe2082ca66f31e286b681de722ac61d8c8ee2f7c037667e8fa447c"}
Oct 30 20:49:26.776186 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:26.776161    4810 generic.go:356] "Generic (PLEG): container finished" podID="41b0e915-e59e-4f23-b4e4-c68bcb4e48e5" containerID="5bb01d8f8ee6e4a7ff8779514a3ceed648fb4b208495736a65c36649bad67381" exitCode=0
Oct 30 20:49:26.776310 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:26.776219    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z" event={"ID":"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5","Type":"ContainerDied","Data":"5bb01d8f8ee6e4a7ff8779514a3ceed648fb4b208495736a65c36649bad67381"}
Oct 30 20:49:28.002420 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:27.986938    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:28.052333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.052262    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:28.131882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.131600    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-xks-ca\") pod \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") "
Oct 30 20:49:28.131882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.131748    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-openshift-ca\") pod \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") "
Oct 30 20:49:28.131882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.131786    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-kube-api-access-mrwnm\" (UniqueName: \"kubernetes.io/projected/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-kube-api-access-mrwnm\") pod \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\" (UID: \"f6aa8b9c-9151-4b39-af65-541dbba0ccd7\") "
Oct 30 20:49:28.137899 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.132332    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-openshift-ca" pod "f6aa8b9c-9151-4b39-af65-541dbba0ccd7" (UID: "f6aa8b9c-9151-4b39-af65-541dbba0ccd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:28.137899 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.134602    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-xks-ca" pod "f6aa8b9c-9151-4b39-af65-541dbba0ccd7" (UID: "f6aa8b9c-9151-4b39-af65-541dbba0ccd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:28.137899 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.134905    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-kube-api-access-mrwnm" pod "f6aa8b9c-9151-4b39-af65-541dbba0ccd7" (UID: "f6aa8b9c-9151-4b39-af65-541dbba0ccd7"). InnerVolumeSpecName "kube-api-access-mrwnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:28.233162 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.233124    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-openshift-ca\") pod \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") "
Oct 30 20:49:28.233162 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.233166    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-kube-api-access-q927r\" (UniqueName: \"kubernetes.io/projected/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-kube-api-access-q927r\") pod \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") "
Oct 30 20:49:28.233538 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.233210    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-xks-ca\") pod \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\" (UID: \"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5\") "
Oct 30 20:49:28.233538 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.233509    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-openshift-ca" pod "41b0e915-e59e-4f23-b4e4-c68bcb4e48e5" (UID: "41b0e915-e59e-4f23-b4e4-c68bcb4e48e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:28.233666 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.233615    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:28.233666 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.233643    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:28.233666 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.233657    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrwnm\" (UniqueName: \"kubernetes.io/projected/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-kube-api-access-mrwnm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:28.233830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.233671    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6aa8b9c-9151-4b39-af65-541dbba0ccd7-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:28.235289 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.235262    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-kube-api-access-q927r" pod "41b0e915-e59e-4f23-b4e4-c68bcb4e48e5" (UID: "41b0e915-e59e-4f23-b4e4-c68bcb4e48e5"). InnerVolumeSpecName "kube-api-access-q927r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:28.235768 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.235747    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-xks-ca" pod "41b0e915-e59e-4f23-b4e4-c68bcb4e48e5" (UID: "41b0e915-e59e-4f23-b4e4-c68bcb4e48e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:28.334754 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.334677    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q927r\" (UniqueName: \"kubernetes.io/projected/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-kube-api-access-q927r\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:28.334754 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.334718    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:28.794903 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.794873    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"
Oct 30 20:49:28.794903 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.794889    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj" event={"ID":"f6aa8b9c-9151-4b39-af65-541dbba0ccd7","Type":"ContainerDied","Data":"df966a96d6ab2a4b203e6cb8148968de3efe8b13b8666f1b90d8eb0c271dedee"}
Oct 30 20:49:28.795158 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.794940    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df966a96d6ab2a4b203e6cb8148968de3efe8b13b8666f1b90d8eb0c271dedee"
Oct 30 20:49:28.796501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.796470    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z" event={"ID":"41b0e915-e59e-4f23-b4e4-c68bcb4e48e5","Type":"ContainerDied","Data":"785bc65b9693ba7a257e8240fefec5f67b3312f7309903f0bedcf2d6989fbffd"}
Oct 30 20:49:28.796628 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.796503    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"
Oct 30 20:49:28.796628 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:28.796521    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="785bc65b9693ba7a257e8240fefec5f67b3312f7309903f0bedcf2d6989fbffd"
Oct 30 20:49:34.968454 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:34.968414    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:49:35.971485 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.971451    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"]
Oct 30 20:49:35.971992 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.971969    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f6aa8b9c-9151-4b39-af65-541dbba0ccd7" containerName="revoke-keys"
Oct 30 20:49:35.971992 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.971988    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6aa8b9c-9151-4b39-af65-541dbba0ccd7" containerName="revoke-keys"
Oct 30 20:49:35.971992 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.971998    4810 cpu_manager.go:414] "containerMap: removing container" podUID="41b0e915-e59e-4f23-b4e4-c68bcb4e48e5" containerName="revoke-keys"
Oct 30 20:49:35.972268 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.972003    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b0e915-e59e-4f23-b4e4-c68bcb4e48e5" containerName="revoke-keys"
Oct 30 20:49:35.972268 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.972116    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b0e915-e59e-4f23-b4e4-c68bcb4e48e5" containerName="revoke-keys"
Oct 30 20:49:35.972268 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.972127    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6aa8b9c-9151-4b39-af65-541dbba0ccd7" containerName="revoke-keys"
Oct 30 20:49:35.976465 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.976443    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:35.983961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.983938    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:49:35.984144 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.984128    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:49:35.986593 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:35.986568    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"]
Oct 30 20:49:36.012678 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.012649    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmzl\" (UniqueName: \"kubernetes.io/projected/df985564-7f1b-4772-84c7-f3fb9486b6dd-kube-api-access-mjmzl\") pod \"maas-api-revoke-sub-3e14b6ff-4zdzj\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:36.012816 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.012742    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e14b6ff-4zdzj\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:36.012940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.012836    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e14b6ff-4zdzj\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:36.113718 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.113686    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e14b6ff-4zdzj\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:36.113913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.113771    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e14b6ff-4zdzj\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:36.113913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.113830    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmzl\" (UniqueName: \"kubernetes.io/projected/df985564-7f1b-4772-84c7-f3fb9486b6dd-kube-api-access-mjmzl\") pod \"maas-api-revoke-sub-3e14b6ff-4zdzj\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:36.114629 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.114604    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e14b6ff-4zdzj\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:36.116044 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.116022    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e14b6ff-4zdzj\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:36.123786 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.123755    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmzl\" (UniqueName: \"kubernetes.io/projected/df985564-7f1b-4772-84c7-f3fb9486b6dd-kube-api-access-mjmzl\") pod \"maas-api-revoke-sub-3e14b6ff-4zdzj\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:36.291894 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.291783    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:36.596497 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.579305    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"]
Oct 30 20:49:36.844052 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:36.844019    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj" event={"ID":"df985564-7f1b-4772-84c7-f3fb9486b6dd","Type":"ContainerStarted","Data":"24abe4e8232b9a2fd2cdbec1d5353a681107eeaef1b367e08bc01b31dbcd314a"}
Oct 30 20:49:37.261980 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.261909    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"]
Oct 30 20:49:37.266514 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.266491    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.285327 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.284752    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"]
Oct 30 20:49:37.328873 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.328820    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvv2z\" (UniqueName: \"kubernetes.io/projected/9e300ef6-e571-4178-9b9b-b7a558122275-kube-api-access-qvv2z\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.329033 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.328898    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e300ef6-e571-4178-9b9b-b7a558122275-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.329097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.329038    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.329097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.329077    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.329261 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.329238    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.329325 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.329277    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.329379 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.329340    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.329437 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.329382    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e300ef6-e571-4178-9b9b-b7a558122275-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.329437 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.329412    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e300ef6-e571-4178-9b9b-b7a558122275-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.430045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430015    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e300ef6-e571-4178-9b9b-b7a558122275-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.430231 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430056    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e300ef6-e571-4178-9b9b-b7a558122275-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.430231 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430127    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvv2z\" (UniqueName: \"kubernetes.io/projected/9e300ef6-e571-4178-9b9b-b7a558122275-kube-api-access-qvv2z\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.430231 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430167    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e300ef6-e571-4178-9b9b-b7a558122275-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.430231 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430219    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.430430 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430242    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.430430 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430314    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.430430 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430338    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.430430 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430366    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.431345 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430723    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.431345 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430833    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e300ef6-e571-4178-9b9b-b7a558122275-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.431345 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.430993    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.431345 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.431083    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.432070 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.432044    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.433604 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.433579    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e300ef6-e571-4178-9b9b-b7a558122275-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.434042 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.434020    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e300ef6-e571-4178-9b9b-b7a558122275-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.439538 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.439517    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e300ef6-e571-4178-9b9b-b7a558122275-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.439943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.439924    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvv2z\" (UniqueName: \"kubernetes.io/projected/9e300ef6-e571-4178-9b9b-b7a558122275-kube-api-access-qvv2z\") pod \"maas-default-gateway-openshift-default-77f556d647-f8lmw\" (UID: \"9e300ef6-e571-4178-9b9b-b7a558122275\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.584596 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.584514    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:37.853491 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.853413    4810 generic.go:356] "Generic (PLEG): container finished" podID="df985564-7f1b-4772-84c7-f3fb9486b6dd" containerID="be2382cf46855d86f4d3e9daa4a298c1d2188742fc0b687a288f3ce8c8952a29" exitCode=0
Oct 30 20:49:37.853642 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.853489    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj" event={"ID":"df985564-7f1b-4772-84c7-f3fb9486b6dd","Type":"ContainerDied","Data":"be2382cf46855d86f4d3e9daa4a298c1d2188742fc0b687a288f3ce8c8952a29"}
Oct 30 20:49:37.883213 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.883187    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"]
Oct 30 20:49:37.888275 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:49:37.888250    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e300ef6_e571_4178_9b9b_b7a558122275.slice/crio-9ddf451ccac36102868d06ef18729eed931057c7fd7c5f53c3691fdb8c141739 WatchSource:0}: Error finding container 9ddf451ccac36102868d06ef18729eed931057c7fd7c5f53c3691fdb8c141739: Status 404 returned error can't find the container with id 9ddf451ccac36102868d06ef18729eed931057c7fd7c5f53c3691fdb8c141739
Oct 30 20:49:37.890719 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.890689    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:37.890788 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.890759    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:37.890834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:37.890787    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:38.861905 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:38.861844    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" event={"ID":"9e300ef6-e571-4178-9b9b-b7a558122275","Type":"ContainerStarted","Data":"908de54b304ad74e13a4224442f950cc89fb67f9e26f0310407eaa6d58e20f4d"}
Oct 30 20:49:38.862393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:38.861942    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" event={"ID":"9e300ef6-e571-4178-9b9b-b7a558122275","Type":"ContainerStarted","Data":"9ddf451ccac36102868d06ef18729eed931057c7fd7c5f53c3691fdb8c141739"}
Oct 30 20:49:38.963253 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:38.963194    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" podStartSLOduration=1.963176083 podStartE2EDuration="1.963176083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:38.891176517 +0000 UTC m=+745.382993067" watchObservedRunningTime="2026-10-30 20:49:38.963176083 +0000 UTC m=+745.454992628"
Oct 30 20:49:38.965435 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:38.965404    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"]
Oct 30 20:49:38.992219 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:38.970714    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:38.999988 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:38.999031    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"]
Oct 30 20:49:39.010458 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.007223    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-317a33-openshift-default-dockercfg-9qjvh\""
Oct 30 20:49:39.050965 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.050933    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.051115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.050977    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.051115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.051056    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.051198 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.051172    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zts\" (UniqueName: \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-kube-api-access-t7zts\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.051243 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.051230    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5475c236-1e38-45ef-b483-9001b2fbf2f4-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.051334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.051320    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.051394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.051373    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.051425 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.051417    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.051466 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.051435    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.112513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.112448    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:39.152964 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.152922    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-xks-ca\") pod \"df985564-7f1b-4772-84c7-f3fb9486b6dd\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") "
Oct 30 20:49:39.153112 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.152977    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df985564-7f1b-4772-84c7-f3fb9486b6dd-kube-api-access-mjmzl\" (UniqueName: \"kubernetes.io/projected/df985564-7f1b-4772-84c7-f3fb9486b6dd-kube-api-access-mjmzl\") pod \"df985564-7f1b-4772-84c7-f3fb9486b6dd\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") "
Oct 30 20:49:39.153112 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153019    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-openshift-ca\") pod \"df985564-7f1b-4772-84c7-f3fb9486b6dd\" (UID: \"df985564-7f1b-4772-84c7-f3fb9486b6dd\") "
Oct 30 20:49:39.153260 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153241    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.153314 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153288    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.153362 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153322    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.153362 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153348    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.153472 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153405    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.153472 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153428    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.153472 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153461    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.153618 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153567    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zts\" (UniqueName: \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-kube-api-access-t7zts\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.153678 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153636    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5475c236-1e38-45ef-b483-9001b2fbf2f4-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.153727 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.153694    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.154287 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.154231    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.154287 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.154232    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.154489 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.154467    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5475c236-1e38-45ef-b483-9001b2fbf2f4-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.154560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.154539    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.154773 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.154754    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-openshift-ca" pod "df985564-7f1b-4772-84c7-f3fb9486b6dd" (UID: "df985564-7f1b-4772-84c7-f3fb9486b6dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:39.156829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.156799    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df985564-7f1b-4772-84c7-f3fb9486b6dd-kube-api-access-mjmzl" pod "df985564-7f1b-4772-84c7-f3fb9486b6dd" (UID: "df985564-7f1b-4772-84c7-f3fb9486b6dd"). InnerVolumeSpecName "kube-api-access-mjmzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:39.157095 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.157075    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.157523 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.157506    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.157748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.157732    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-xks-ca" pod "df985564-7f1b-4772-84c7-f3fb9486b6dd" (UID: "df985564-7f1b-4772-84c7-f3fb9486b6dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:39.163784 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.163762    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zts\" (UniqueName: \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-kube-api-access-t7zts\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.164362 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.164330    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.256882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.254496    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:39.256882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.254529    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjmzl\" (UniqueName: \"kubernetes.io/projected/df985564-7f1b-4772-84c7-f3fb9486b6dd-kube-api-access-mjmzl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:39.256882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.254541    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df985564-7f1b-4772-84c7-f3fb9486b6dd-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:49:39.317647 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.317616    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:39.603938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.597505    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:39.603938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.597632    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f8lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:49:39.603938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.597675    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" podUID="9e300ef6-e571-4178-9b9b-b7a558122275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 20:49:39.631215 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.631191    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"]
Oct 30 20:49:39.633720 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:49:39.633694    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5475c236_1e38_45ef_b483_9001b2fbf2f4.slice/crio-8d266c44d48d44afe4c25daabe52468f725805aaee373a484878db16156faa43 WatchSource:0}: Error finding container 8d266c44d48d44afe4c25daabe52468f725805aaee373a484878db16156faa43: Status 404 returned error can't find the container with id 8d266c44d48d44afe4c25daabe52468f725805aaee373a484878db16156faa43
Oct 30 20:49:39.636601 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.636568    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:39.636677 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.636634    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:39.636677 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.636661    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:39.869343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.869315    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" event={"ID":"5475c236-1e38-45ef-b483-9001b2fbf2f4","Type":"ContainerStarted","Data":"04815a806bc94d7c09edae61cdc2be5886ff138f6c1aece93382909172c65dd3"}
Oct 30 20:49:39.869716 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.869364    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" event={"ID":"5475c236-1e38-45ef-b483-9001b2fbf2f4","Type":"ContainerStarted","Data":"8d266c44d48d44afe4c25daabe52468f725805aaee373a484878db16156faa43"}
Oct 30 20:49:39.870569 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.870545    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"
Oct 30 20:49:39.870644 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.870575    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj" event={"ID":"df985564-7f1b-4772-84c7-f3fb9486b6dd","Type":"ContainerDied","Data":"24abe4e8232b9a2fd2cdbec1d5353a681107eeaef1b367e08bc01b31dbcd314a"}
Oct 30 20:49:39.870644 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.870605    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24abe4e8232b9a2fd2cdbec1d5353a681107eeaef1b367e08bc01b31dbcd314a"
Oct 30 20:49:39.902473 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:39.894672    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podStartSLOduration=1.894656087 podStartE2EDuration="1.894656087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:39.892569083 +0000 UTC m=+746.384385649" watchObservedRunningTime="2026-10-30 20:49:39.894656087 +0000 UTC m=+746.386472632"
Oct 30 20:49:40.318872 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:40.318775    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:40.320249 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:40.320222    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:49:40.320386 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:40.320270    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 20:49:40.594314 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:40.594224    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f8lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:49:40.594583 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:40.594551    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" podUID="9e300ef6-e571-4178-9b9b-b7a558122275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 20:49:41.322948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:41.322912    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:49:41.323545 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:41.323515    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 20:49:41.586394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:41.586296    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f8lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:49:41.586781 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:41.586736    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" podUID="9e300ef6-e571-4178-9b9b-b7a558122275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 20:49:42.326126 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:42.325987    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:49:42.326126 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:42.326062    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 20:49:42.585313 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:42.585227    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f8lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:49:42.585558 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:42.585526    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" podUID="9e300ef6-e571-4178-9b9b-b7a558122275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 20:49:43.318266 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:43.318226    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:49:43.318564 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:43.318533    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 20:49:43.588948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:43.588557    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f8lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:49:43.588948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:43.588632    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" podUID="9e300ef6-e571-4178-9b9b-b7a558122275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 20:49:44.318265 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:44.318222    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:49:44.318473 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:44.318287    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 20:49:44.585217 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:44.585133    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f8lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:49:44.585461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:44.585438    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" podUID="9e300ef6-e571-4178-9b9b-b7a558122275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 20:49:45.318503 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:45.318469    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:49:45.318978 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:45.318537    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 20:49:45.589934 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:45.588306    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f8lmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:49:45.589934 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:45.588383    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" podUID="9e300ef6-e571-4178-9b9b-b7a558122275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 30 20:49:46.318972 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.318932    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:49:46.319339 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.318996    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 20:49:46.637066 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.637037    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:46.637888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.637375    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:46.638571 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.638542    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw"
Oct 30 20:49:46.692220 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.692185    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"]
Oct 30 20:49:46.692971 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.692949    4810 cpu_manager.go:414] "containerMap: removing container" podUID="df985564-7f1b-4772-84c7-f3fb9486b6dd" containerName="revoke-keys"
Oct 30 20:49:46.693081 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.692972    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df985564-7f1b-4772-84c7-f3fb9486b6dd" containerName="revoke-keys"
Oct 30 20:49:46.693139 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.693090    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="df985564-7f1b-4772-84c7-f3fb9486b6dd" containerName="revoke-keys"
Oct 30 20:49:46.698186 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.698163    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.707108 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.707069    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f9a819-openshift-default-dockercfg-925js\""
Oct 30 20:49:46.713020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.712998    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"]
Oct 30 20:49:46.799189 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.799158    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.799360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.799194    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20525970-334f-4fe4-984a-1f6c89329c3f-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.799360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.799236    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.799360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.799342    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20525970-334f-4fe4-984a-1f6c89329c3f-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.799530 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.799438    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6zl9\" (UniqueName: \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-kube-api-access-j6zl9\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.799584 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.799539    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.799631 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.799579    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.799679 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.799624    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.799792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.799777    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.900974 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.900870    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.900974 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.900931    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.900974 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.900958    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.901261 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.901003    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.901261 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.901045    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.901382 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.901349    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.901446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.901396    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.901446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.901407    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20525970-334f-4fe4-984a-1f6c89329c3f-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.901547 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.901445    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.901547 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.901503    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.901547 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.901535    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20525970-334f-4fe4-984a-1f6c89329c3f-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.901688 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.901571    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6zl9\" (UniqueName: \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-kube-api-access-j6zl9\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.901834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.901810    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.902290 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.902265    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20525970-334f-4fe4-984a-1f6c89329c3f-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.904207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.904112    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.904353 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.904330    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20525970-334f-4fe4-984a-1f6c89329c3f-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.910943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.910919    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:46.911130 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:46.911010    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6zl9\" (UniqueName: \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-kube-api-access-j6zl9\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:47.020963 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.020921    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:47.325455 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:49:47.325425    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20525970_334f_4fe4_984a_1f6c89329c3f.slice/crio-11022623c495ae0e7c6336d399c79d53d47aca5e24ce7b01fbae2d025caa5185 WatchSource:0}: Error finding container 11022623c495ae0e7c6336d399c79d53d47aca5e24ce7b01fbae2d025caa5185: Status 404 returned error can't find the container with id 11022623c495ae0e7c6336d399c79d53d47aca5e24ce7b01fbae2d025caa5185
Oct 30 20:49:47.325777 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.325679    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"]
Oct 30 20:49:47.328265 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.328231    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:47.328368 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.328303    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:47.328368 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.328342    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:47.407835 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.406848    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:47.407835 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.407484    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:47.410323 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.410275    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 20:49:47.971778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.971738    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" event={"ID":"20525970-334f-4fe4-984a-1f6c89329c3f","Type":"ContainerStarted","Data":"f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30"}
Oct 30 20:49:47.971985 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.971803    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" event={"ID":"20525970-334f-4fe4-984a-1f6c89329c3f","Type":"ContainerStarted","Data":"11022623c495ae0e7c6336d399c79d53d47aca5e24ce7b01fbae2d025caa5185"}
Oct 30 20:49:47.998558 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:47.998501    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" podStartSLOduration=1.9984838169999999 podStartE2EDuration="1.998483817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:47.995735832 +0000 UTC m=+754.487552378" watchObservedRunningTime="2026-10-30 20:49:47.998483817 +0000 UTC m=+754.490300364"
Oct 30 20:49:48.021730 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:48.021683    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:48.022731 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:48.022694    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:49:48.022889 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:48.022747    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 20:49:48.603275 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:48.603231    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:49:49.021493 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:49.021458    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:49:49.021670 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:49.021521    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 20:49:50.022281 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.022248    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:49:50.022745 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.022301    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 20:49:50.382310 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.382283    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"]
Oct 30 20:49:50.400736 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.400093    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.406165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.406141    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-317a33\"/\"distinct-w1-317a33-kserve-self-signed-certs\""
Oct 30 20:49:50.406941 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.406332    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"]
Oct 30 20:49:50.446699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.446669    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-home\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.446924 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.446904    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqcp\" (UniqueName: \"kubernetes.io/projected/34915455-2f8c-46e5-ac31-89605f8ffe94-kube-api-access-bkqcp\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.447095 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.447079    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-dshm\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.447230 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.447215    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-model-cache\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.447337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.447325    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34915455-2f8c-46e5-ac31-89605f8ffe94-tls-certs\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.447462 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.447449    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-tmp-dir\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.549081 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.548227    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34915455-2f8c-46e5-ac31-89605f8ffe94-tls-certs\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.549081 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.548583    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-tmp-dir\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.549081 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.548698    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-home\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.549081 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.548725    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqcp\" (UniqueName: \"kubernetes.io/projected/34915455-2f8c-46e5-ac31-89605f8ffe94-kube-api-access-bkqcp\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.549081 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.548793    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-dshm\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.549081 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.548850    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-model-cache\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.549514 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.549131    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-tmp-dir\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.549514 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.549188    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-model-cache\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.549623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.549578    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-home\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.552420 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.552264    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34915455-2f8c-46e5-ac31-89605f8ffe94-tls-certs\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.555883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.555842    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-dshm\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.560267 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.560222    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqcp\" (UniqueName: \"kubernetes.io/projected/34915455-2f8c-46e5-ac31-89605f8ffe94-kube-api-access-bkqcp\") pod \"distinct-w1-317a33-kserve-5f6f556978-msxc9\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:50.725131 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:50.725058    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:51.022082 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:51.022015    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:49:51.022082 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:51.022065    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 20:49:51.037921 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:51.037898    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"]
Oct 30 20:49:51.039752 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:49:51.039726    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34915455_2f8c_46e5_ac31_89605f8ffe94.slice/crio-33b53dfbff47c61998ac33abff7f611672cc89056e25c1c2a524152573ee5043 WatchSource:0}: Error finding container 33b53dfbff47c61998ac33abff7f611672cc89056e25c1c2a524152573ee5043: Status 404 returned error can't find the container with id 33b53dfbff47c61998ac33abff7f611672cc89056e25c1c2a524152573ee5043
Oct 30 20:49:52.021965 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.021931    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:49:52.022203 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.022173    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 20:49:52.027063 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.027029    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" event={"ID":"34915455-2f8c-46e5-ac31-89605f8ffe94","Type":"ContainerStarted","Data":"ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d"}
Oct 30 20:49:52.027170 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.027079    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" event={"ID":"34915455-2f8c-46e5-ac31-89605f8ffe94","Type":"ContainerStarted","Data":"33b53dfbff47c61998ac33abff7f611672cc89056e25c1c2a524152573ee5043"}
Oct 30 20:49:52.048229 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.048174    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podStartSLOduration=2.048157743 podStartE2EDuration="2.048157743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:52.046110625 +0000 UTC m=+758.537927166" watchObservedRunningTime="2026-10-30 20:49:52.048157743 +0000 UTC m=+758.539974289"
Oct 30 20:49:52.391877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.389303    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"]
Oct 30 20:49:52.409143 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.409115    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.420131 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.420097    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"]
Oct 30 20:49:52.517278 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.517242    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b83400-6692-4ba7-b59f-14c215517039-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.517278 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.517283    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.517469 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.517362    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.517469 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.517412    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b83400-6692-4ba7-b59f-14c215517039-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.517469 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.517441    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99plp\" (UniqueName: \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-kube-api-access-99plp\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.517565 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.517471    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.517565 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.517518    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.517565 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.517555    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.517650 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.517571    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.618746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.618705    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.618925 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.618764    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.618925 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.618786    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.618925 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.618894    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b83400-6692-4ba7-b59f-14c215517039-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.618925 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.618923    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.619080 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.618966    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.619080 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.619023    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b83400-6692-4ba7-b59f-14c215517039-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.619080 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.619068    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99plp\" (UniqueName: \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-kube-api-access-99plp\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.619170 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.619109    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.619250 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.619216    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.619321 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.619266    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.619430 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.619411    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.619969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.619907    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b83400-6692-4ba7-b59f-14c215517039-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.619969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.619922    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.621918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.621897    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b83400-6692-4ba7-b59f-14c215517039-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.622009 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.621934    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.629201 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.629077    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.629451 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.629420    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99plp\" (UniqueName: \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-kube-api-access-99plp\") pod \"maas-default-gateway-openshift-default-77f556d647-cwl6l\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:52.725914 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.725825    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:49:52.736105 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:52.736075    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:53.022005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:53.021905    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:49:53.022005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:53.021968    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 20:49:53.056524 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:53.056500    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"]
Oct 30 20:49:53.058478 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:49:53.058454    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b83400_6692_4ba7_b59f_14c215517039.slice/crio-a837e0e80aaa8260a09c92006e499245e1e277c04b3c45d249db4cc502042df8 WatchSource:0}: Error finding container a837e0e80aaa8260a09c92006e499245e1e277c04b3c45d249db4cc502042df8: Status 404 returned error can't find the container with id a837e0e80aaa8260a09c92006e499245e1e277c04b3c45d249db4cc502042df8
Oct 30 20:49:53.060774 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:53.060744    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:53.060891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:53.060800    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:53.060891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:53.060826    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:54.024502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:54.021539    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:49:54.024502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:54.021594    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 20:49:54.051605 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:54.049778    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" event={"ID":"16b83400-6692-4ba7-b59f-14c215517039","Type":"ContainerStarted","Data":"15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6"}
Oct 30 20:49:54.051605 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:54.049821    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" event={"ID":"16b83400-6692-4ba7-b59f-14c215517039","Type":"ContainerStarted","Data":"a837e0e80aaa8260a09c92006e499245e1e277c04b3c45d249db4cc502042df8"}
Oct 30 20:49:54.138883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:54.133752    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podStartSLOduration=2.133732891 podStartE2EDuration="2.133732891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:54.075457812 +0000 UTC m=+760.567274356" watchObservedRunningTime="2026-10-30 20:49:54.133732891 +0000 UTC m=+760.625549437"
Oct 30 20:49:54.736639 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:54.736563    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:49:54.737555 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:54.737537    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cwl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:49:54.737605 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:54.737576    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:49:55.025623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:55.025544    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:55.025818 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:55.025798    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:55.026573 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:55.026553    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 20:49:55.162471 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:49:55.162436    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:49:55.736989 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:55.736953    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cwl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:49:55.737144 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:55.737001    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:49:55.992935 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:55.987247    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"]
Oct 30 20:49:55.992935 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:55.992158    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.016878 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.016435    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-bc102e-openshift-default-dockercfg-wm7tq\""
Oct 30 20:49:56.018244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.018208    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"]
Oct 30 20:49:56.068330 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.068298    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.068490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.068340    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmt5t\" (UniqueName: \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-kube-api-access-cmt5t\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.068490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.068448    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.068559 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.068495    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.068559 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.068532    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.068627 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.068614    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.068663 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.068637    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.068702 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.068677    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.068744 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.068731    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.169695 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.169664    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.170172 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.169742    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.170172 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.169780    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmt5t\" (UniqueName: \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-kube-api-access-cmt5t\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.170172 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.169833    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.170172 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.169901    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.170172 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.169972    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.170580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.170551    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.192795 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.192764    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.193050 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.193029    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.210884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.196389    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.210884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.209886    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.210884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.196549    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.210884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.197469    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.210884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.196625    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.213229 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.212713    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.215454 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.215436    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.215589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.215438    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.215589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.215484    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmt5t\" (UniqueName: \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-kube-api-access-cmt5t\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.332017 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.331928    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:56.658448 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.658417    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"]
Oct 30 20:49:56.660363 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:49:56.660334    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6d65a3_e831_4dd3_bc81_1121b26717c2.slice/crio-24ed3932357c0a8edc93f9cf0c81565a53171b4f097377113c9979ad1ead738c WatchSource:0}: Error finding container 24ed3932357c0a8edc93f9cf0c81565a53171b4f097377113c9979ad1ead738c: Status 404 returned error can't find the container with id 24ed3932357c0a8edc93f9cf0c81565a53171b4f097377113c9979ad1ead738c
Oct 30 20:49:56.663006 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.662967    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:56.663066 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.663051    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:56.663117 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.663101    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:49:56.737700 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.737670    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cwl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:49:56.737810 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:56.737727    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:49:57.076970 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:57.071843    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" event={"ID":"4e6d65a3-e831-4dd3-bc81-1121b26717c2","Type":"ContainerStarted","Data":"b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e"}
Oct 30 20:49:57.076970 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:57.071923    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" event={"ID":"4e6d65a3-e831-4dd3-bc81-1121b26717c2","Type":"ContainerStarted","Data":"24ed3932357c0a8edc93f9cf0c81565a53171b4f097377113c9979ad1ead738c"}
Oct 30 20:49:57.179883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:57.179686    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" podStartSLOduration=2.179655102 podStartE2EDuration="2.179655102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:57.178380266 +0000 UTC m=+763.670196821" watchObservedRunningTime="2026-10-30 20:49:57.179655102 +0000 UTC m=+763.671471647"
Oct 30 20:49:57.337948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:57.337597    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:49:57.337948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:57.337764    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:49:57.337948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:57.337809    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:49:57.740885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:57.740238    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cwl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:49:57.740885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:57.740298    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:49:58.333882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:58.333618    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:49:58.333882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:58.333679    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:49:58.741279 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:58.741234    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cwl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:49:58.741490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:58.741299    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:49:59.336884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:59.334283    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:49:59.336884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:59.334357    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:49:59.747269 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:59.747234    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cwl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:49:59.747590 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:49:59.747563    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:50:00.335889 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:00.335238    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:50:00.335889 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:00.335316    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:50:00.730751 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:00.730722    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:50:00.738257 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:00.738228    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cwl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:50:00.738452 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:00.738427    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:50:01.000950 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:00.994249    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:50:01.233148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:01.233117    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:50:01.332957 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:01.332842    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:50:01.333209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:01.333172    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:50:01.737355 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:01.737318    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cwl6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:50:01.737733 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:01.737412    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 20:50:02.333312 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:02.333271    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:50:02.333490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:02.333357    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:50:03.120905 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:03.119914    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:50:03.121399 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:03.120959    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:50:03.121592 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:03.121572    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 20:50:03.332779 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:03.332746    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:50:03.333040 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:03.333019    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 20:50:04.817555 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:04.817521    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:50:04.820582 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:04.820551    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:50:04.822392 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:04.822372    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 20:50:07.421069 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.421036    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:50:07.583526 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.583493    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"]
Oct 30 20:50:07.589937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.589913    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.624477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.624446    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-317a33\"/\"distinct-2-w1-317a33-kserve-self-signed-certs\""
Oct 30 20:50:07.638037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.638008    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"]
Oct 30 20:50:07.638400 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.638346    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-model-cache\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.638400 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.638396    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-tmp-dir\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.638577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.638429    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71e8b54-b7bf-4df7-bd10-186443f455ee-tls-certs\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.638630 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.638571    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-dshm\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.638683 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.638643    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lf7g\" (UniqueName: \"kubernetes.io/projected/c71e8b54-b7bf-4df7-bd10-186443f455ee-kube-api-access-4lf7g\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.638683 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.638675    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-home\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.739285 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.739196    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lf7g\" (UniqueName: \"kubernetes.io/projected/c71e8b54-b7bf-4df7-bd10-186443f455ee-kube-api-access-4lf7g\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.739285 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.739234    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-home\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.739285 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.739270    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-model-cache\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.739556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.739293    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-tmp-dir\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.739556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.739318    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71e8b54-b7bf-4df7-bd10-186443f455ee-tls-certs\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.739556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.739480    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-dshm\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.739799 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.739775    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-home\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.739955 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.739936    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-model-cache\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.740290 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.740266    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-tmp-dir\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.741776 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.741748    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-dshm\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.742310 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.742287    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71e8b54-b7bf-4df7-bd10-186443f455ee-tls-certs\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.776034 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.776009    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lf7g\" (UniqueName: \"kubernetes.io/projected/c71e8b54-b7bf-4df7-bd10-186443f455ee-kube-api-access-4lf7g\") pod \"distinct-2-w1-317a33-kserve-7486c95665-6xjmx\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") " pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:07.904061 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:07.904016    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:08.220564 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:08.220531    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"]
Oct 30 20:50:08.222232 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:50:08.222201    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc71e8b54_b7bf_4df7_bd10_186443f455ee.slice/crio-d7369f54d092be15930f61585c97f2a4e05a1cb6360c469661550d52426a573c WatchSource:0}: Error finding container d7369f54d092be15930f61585c97f2a4e05a1cb6360c469661550d52426a573c: Status 404 returned error can't find the container with id d7369f54d092be15930f61585c97f2a4e05a1cb6360c469661550d52426a573c
Oct 30 20:50:08.272932 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:08.272887    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" event={"ID":"c71e8b54-b7bf-4df7-bd10-186443f455ee","Type":"ContainerStarted","Data":"d7369f54d092be15930f61585c97f2a4e05a1cb6360c469661550d52426a573c"}
Oct 30 20:50:09.281768 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:09.281728    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" event={"ID":"c71e8b54-b7bf-4df7-bd10-186443f455ee","Type":"ContainerStarted","Data":"8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31"}
Oct 30 20:50:09.310523 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:09.310466    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podStartSLOduration=2.310452068 podStartE2EDuration="2.310452068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:09.306805532 +0000 UTC m=+775.798622072" watchObservedRunningTime="2026-10-30 20:50:09.310452068 +0000 UTC m=+775.802268611"
Oct 30 20:50:09.906988 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:09.906932    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:17.127878 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.127661    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"]
Oct 30 20:50:17.138573 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.138544    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.144647 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.144621    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"]
Oct 30 20:50:17.158603 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.158576    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bc102e\"/\"e2e-conflict-b-b39ef5b3-kserve-self-signed-certs\""
Oct 30 20:50:17.215825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.215730    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tmp-dir\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.216032 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.215851    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-home\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.216032 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.215970    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6k8c\" (UniqueName: \"kubernetes.io/projected/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-kube-api-access-f6k8c\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.216158 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.216044    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tls-certs\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.216217 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.216206    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-model-cache\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.216353 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.216301    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-dshm\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.317629 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.317586    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6k8c\" (UniqueName: \"kubernetes.io/projected/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-kube-api-access-f6k8c\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.317809 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.317661    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tls-certs\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.317809 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.317772    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-model-cache\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.317911 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.317895    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-dshm\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.317958 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.317948    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tmp-dir\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.317993 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.317985    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-home\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.318475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.318455    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-home\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.322274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.322243    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tmp-dir\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.322385 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.322276    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-model-cache\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.323012 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.322987    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tls-certs\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.324368 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.324346    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-dshm\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.328338 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.328311    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6k8c\" (UniqueName: \"kubernetes.io/projected/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-kube-api-access-f6k8c\") pod \"e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") " pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.456475 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.456393    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:17.742880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.742646    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"]
Oct 30 20:50:17.904275 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.904237    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:17.923313 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:17.923282    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:18.418067 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:18.418027    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr" event={"ID":"cb2b8ac5-f431-4fdb-82df-e8f935e561f7","Type":"ContainerStarted","Data":"8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d"}
Oct 30 20:50:18.418545 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:18.418079    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr" event={"ID":"cb2b8ac5-f431-4fdb-82df-e8f935e561f7","Type":"ContainerStarted","Data":"0d998a1a602df21b9256cf8faf961e7c26127c47fc60ef8b06b0331bc1358c2a"}
Oct 30 20:50:18.429529 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:18.429505    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:50:18.442116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:18.442075    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr" podStartSLOduration=1.442061687 podStartE2EDuration="1.442061687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:18.440205713 +0000 UTC m=+784.932022258" watchObservedRunningTime="2026-10-30 20:50:18.442061687 +0000 UTC m=+784.933878231"
Oct 30 20:50:18.456850 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:18.456827    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:25.155207 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:50:25.155166    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:50:27.456926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:27.456894    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:27.469487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:27.469463    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:27.495249 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:27.495225    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:27.689444 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:27.689412    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:50:31.957833 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:31.957492    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"]
Oct 30 20:50:31.964115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:31.964077    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"]
Oct 30 20:50:31.964265 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:31.964212    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.139661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.126556    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.139661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.126617    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.139661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.126663    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.139661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.126711    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.139661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.126750    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.139661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.126793    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.139661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.126892    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.139661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.126979    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.162923 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.160991    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnmrj\" (UniqueName: \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-kube-api-access-wnmrj\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.262844 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.262756    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.262844 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.262824    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.263125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.262893    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.263125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.262945    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.263125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.263016    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.263125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.263085    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.263125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.263105    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnmrj\" (UniqueName: \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-kube-api-access-wnmrj\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.263422 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.263144    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.263422 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.263163    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.263534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.263478    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.263704 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.263684    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.264274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.264251    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.267011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.266986    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.269313 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.269248    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.269661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.269636    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.276146 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.276118    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.281869 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.281827    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.325106 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.325063    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnmrj\" (UniqueName: \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-kube-api-access-wnmrj\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.351502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.351474    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:32.695478 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.695445    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"]
Oct 30 20:50:32.696042 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:50:32.696017    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4234e57_d237_4b81_b8b6_a1ea7e75e12a.slice/crio-44e4882e6155101e7391258335fbad8918b1f0066fae0c3865ba9caee615c986 WatchSource:0}: Error finding container 44e4882e6155101e7391258335fbad8918b1f0066fae0c3865ba9caee615c986: Status 404 returned error can't find the container with id 44e4882e6155101e7391258335fbad8918b1f0066fae0c3865ba9caee615c986
Oct 30 20:50:32.698485 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.698453    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:50:32.698584 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.698515    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:50:32.698584 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.698546    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:50:32.758239 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:32.758212    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" event={"ID":"f4234e57-d237-4b81-b8b6-a1ea7e75e12a","Type":"ContainerStarted","Data":"44e4882e6155101e7391258335fbad8918b1f0066fae0c3865ba9caee615c986"}
Oct 30 20:50:33.766622 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:33.766582    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" event={"ID":"f4234e57-d237-4b81-b8b6-a1ea7e75e12a","Type":"ContainerStarted","Data":"0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856"}
Oct 30 20:50:33.816556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:33.816503    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" podStartSLOduration=2.816484391 podStartE2EDuration="2.816484391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:33.814941025 +0000 UTC m=+800.306757571" watchObservedRunningTime="2026-10-30 20:50:33.816484391 +0000 UTC m=+800.308300932"
Oct 30 20:50:34.353514 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:34.353483    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:34.355235 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:34.355205    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:50:34.355379 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:34.355269    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:50:35.354037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:35.354001    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:50:35.354448 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:35.354063    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:50:36.354282 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:36.354247    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:50:36.354748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:36.354299    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:50:36.949375 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:36.949344    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"]
Oct 30 20:50:36.953395 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:36.953377    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:36.963131 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:36.963084    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:50:36.963270 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:36.963084    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:50:36.967105 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:36.967082    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"]
Oct 30 20:50:37.043748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.043713    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9b22a6e-4zr72\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:37.043748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.043754    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvf8c\" (UniqueName: \"kubernetes.io/projected/86010140-5aad-4945-a9e7-4c46e180574f-kube-api-access-vvf8c\") pod \"maas-api-revoke-sub-e9b22a6e-4zr72\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:37.044028 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.043927    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9b22a6e-4zr72\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:37.145391 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.145355    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9b22a6e-4zr72\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:37.145602 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.145474    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9b22a6e-4zr72\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:37.145602 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.145494    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvf8c\" (UniqueName: \"kubernetes.io/projected/86010140-5aad-4945-a9e7-4c46e180574f-kube-api-access-vvf8c\") pod \"maas-api-revoke-sub-e9b22a6e-4zr72\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:37.146201 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.146176    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9b22a6e-4zr72\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:37.147886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.147849    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9b22a6e-4zr72\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:37.162833 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.162809    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvf8c\" (UniqueName: \"kubernetes.io/projected/86010140-5aad-4945-a9e7-4c46e180574f-kube-api-access-vvf8c\") pod \"maas-api-revoke-sub-e9b22a6e-4zr72\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:37.269805 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.269727    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:37.353960 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.353913    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:50:37.354080 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.353971    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:50:37.598851 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.598738    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"]
Oct 30 20:50:37.602346 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:50:37.602314    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86010140_5aad_4945_a9e7_4c46e180574f.slice/crio-306aa7092f7537bcb7f1b7c5c2be7d062c8bcfac3223cb3d37466607e82e4398 WatchSource:0}: Error finding container 306aa7092f7537bcb7f1b7c5c2be7d062c8bcfac3223cb3d37466607e82e4398: Status 404 returned error can't find the container with id 306aa7092f7537bcb7f1b7c5c2be7d062c8bcfac3223cb3d37466607e82e4398
Oct 30 20:50:37.824034 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:37.824005    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72" event={"ID":"86010140-5aad-4945-a9e7-4c46e180574f","Type":"ContainerStarted","Data":"306aa7092f7537bcb7f1b7c5c2be7d062c8bcfac3223cb3d37466607e82e4398"}
Oct 30 20:50:38.322962 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:38.322506    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"]
Oct 30 20:50:38.353753 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:38.353700    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:50:38.353957 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:38.353778    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:50:38.834304 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:38.834224    4810 generic.go:356] "Generic (PLEG): container finished" podID="86010140-5aad-4945-a9e7-4c46e180574f" containerID="39bbfc5f747e3fd273643cf499f32583181c083d2d88d334b542d1bb29c10690" exitCode=0
Oct 30 20:50:38.834677 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:38.834312    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72" event={"ID":"86010140-5aad-4945-a9e7-4c46e180574f","Type":"ContainerDied","Data":"39bbfc5f747e3fd273643cf499f32583181c083d2d88d334b542d1bb29c10690"}
Oct 30 20:50:39.354028 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:39.353991    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:50:39.354201 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:39.354047    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 20:50:40.040620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.040594    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:40.084846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.084813    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-xks-ca\") pod \"86010140-5aad-4945-a9e7-4c46e180574f\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") "
Oct 30 20:50:40.085030 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.084909    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-openshift-ca\") pod \"86010140-5aad-4945-a9e7-4c46e180574f\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") "
Oct 30 20:50:40.085085 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.085055    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86010140-5aad-4945-a9e7-4c46e180574f-kube-api-access-vvf8c\" (UniqueName: \"kubernetes.io/projected/86010140-5aad-4945-a9e7-4c46e180574f-kube-api-access-vvf8c\") pod \"86010140-5aad-4945-a9e7-4c46e180574f\" (UID: \"86010140-5aad-4945-a9e7-4c46e180574f\") "
Oct 30 20:50:40.085429 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.085333    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-openshift-ca" pod "86010140-5aad-4945-a9e7-4c46e180574f" (UID: "86010140-5aad-4945-a9e7-4c46e180574f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:40.085590 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.085570    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:40.087120 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.087095    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-xks-ca" pod "86010140-5aad-4945-a9e7-4c46e180574f" (UID: "86010140-5aad-4945-a9e7-4c46e180574f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:50:40.087607 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.087583    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86010140-5aad-4945-a9e7-4c46e180574f-kube-api-access-vvf8c" pod "86010140-5aad-4945-a9e7-4c46e180574f" (UID: "86010140-5aad-4945-a9e7-4c46e180574f"). InnerVolumeSpecName "kube-api-access-vvf8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:40.186894 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.186843    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvf8c\" (UniqueName: \"kubernetes.io/projected/86010140-5aad-4945-a9e7-4c46e180574f-kube-api-access-vvf8c\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:40.186894 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.186894    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86010140-5aad-4945-a9e7-4c46e180574f-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:40.358067 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.357990    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:40.358374 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.358355    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:40.358998 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.358977    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 20:50:40.852896 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.851021    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"
Oct 30 20:50:40.852896 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.851062    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72" event={"ID":"86010140-5aad-4945-a9e7-4c46e180574f","Type":"ContainerDied","Data":"306aa7092f7537bcb7f1b7c5c2be7d062c8bcfac3223cb3d37466607e82e4398"}
Oct 30 20:50:40.852896 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:40.851110    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="306aa7092f7537bcb7f1b7c5c2be7d062c8bcfac3223cb3d37466607e82e4398"
Oct 30 20:50:41.106396 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.103592    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"]
Oct 30 20:50:41.106396 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.104466    4810 cpu_manager.go:414] "containerMap: removing container" podUID="86010140-5aad-4945-a9e7-4c46e180574f" containerName="revoke-keys"
Oct 30 20:50:41.106396 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.104487    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86010140-5aad-4945-a9e7-4c46e180574f" containerName="revoke-keys"
Oct 30 20:50:41.106396 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.104631    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="86010140-5aad-4945-a9e7-4c46e180574f" containerName="revoke-keys"
Oct 30 20:50:41.108901 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.108849    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.130729 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.130699    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"]
Oct 30 20:50:41.217344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.217307    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.217520 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.217347    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf96d91-a658-426b-8ee0-e471f8ef6730-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.217520 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.217457    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.217520 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.217488    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.217685 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.217575    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.217685 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.217644    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.217763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.217704    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5b4x\" (UniqueName: \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-kube-api-access-g5b4x\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.217812 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.217784    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.217891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.217828    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319217 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319183    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319217 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319229    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319267    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319323    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319376    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5b4x\" (UniqueName: \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-kube-api-access-g5b4x\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319439    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319495    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319584    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319612    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf96d91-a658-426b-8ee0-e471f8ef6730-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319638    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319990 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319712    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.319990 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.319763    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.320100 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.320081    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.320328 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.320304    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf96d91-a658-426b-8ee0-e471f8ef6730-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.322433 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.322414    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.322581 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.322561    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.335117 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.335082    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5b4x\" (UniqueName: \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-kube-api-access-g5b4x\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.335506 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.335483    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.426883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.426835    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:41.712352 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.712328    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"]
Oct 30 20:50:41.714398 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:50:41.714370    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf96d91_a658_426b_8ee0_e471f8ef6730.slice/crio-db79764c459fb7c6c83f3855818905614ba682e9ccef705499a55907da60e984 WatchSource:0}: Error finding container db79764c459fb7c6c83f3855818905614ba682e9ccef705499a55907da60e984: Status 404 returned error can't find the container with id db79764c459fb7c6c83f3855818905614ba682e9ccef705499a55907da60e984
Oct 30 20:50:41.717299 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.717268    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:50:41.717389 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.717329    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:50:41.717389 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.717359    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:50:41.879877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.879572    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" event={"ID":"bcf96d91-a658-426b-8ee0-e471f8ef6730","Type":"ContainerStarted","Data":"f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1"}
Oct 30 20:50:41.879877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:41.879712    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" event={"ID":"bcf96d91-a658-426b-8ee0-e471f8ef6730","Type":"ContainerStarted","Data":"db79764c459fb7c6c83f3855818905614ba682e9ccef705499a55907da60e984"}
Oct 30 20:50:42.428254 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:42.427926    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:42.429916 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:42.429850    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:50:42.430064 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:42.429945    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:50:43.427556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:43.427518    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:50:43.427772 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:43.427583    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:50:44.428165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:44.428131    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:50:44.428550 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:44.428184    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:50:45.434882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:45.431557    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:50:45.434882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:45.431618    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:50:46.516526 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:46.516470    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:50:46.517187 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:46.517147    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:50:47.251886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.251772    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podStartSLOduration=6.251750846 podStartE2EDuration="6.251750846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:41.911273565 +0000 UTC m=+808.403090109" watchObservedRunningTime="2026-10-30 20:50:47.251750846 +0000 UTC m=+813.743567391"
Oct 30 20:50:47.260076 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.260040    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"]
Oct 30 20:50:47.306371 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.297243    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.307996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.307970    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"]
Oct 30 20:50:47.316483 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.315925    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 20:50:47.316483 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.315994    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 20:50:47.316483 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.316040    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 20:50:47.316483 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.316177    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gjgsg\""
Oct 30 20:50:47.364892 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.354973    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:47.466131 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.438991    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:50:47.466131 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.439101    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:50:47.564597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.346874    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.564597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.562834    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.564597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.562906    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.564597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.562995    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.564597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.563067    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.564597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.563121    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.564597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.563167    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcbjs\" (UniqueName: \"kubernetes.io/projected/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-kube-api-access-xcbjs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.564597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.563946    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:47.671573 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.671529    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.671950 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.671923    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.672118 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.672097    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.672302 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.672285    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.672468 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.672450    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.672603 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.672587    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.685744 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.685707    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcbjs\" (UniqueName: \"kubernetes.io/projected/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-kube-api-access-xcbjs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.686384 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.679707    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.748307 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.747982    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:47.807886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.783738    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.807886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.785277    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.807886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.785611    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.808805 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.808737    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.809251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.809228    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:47.854940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:47.845888    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcbjs\" (UniqueName: \"kubernetes.io/projected/e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33-kube-api-access-xcbjs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k2n4q\" (UID: \"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:48.021883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.021378    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:48.400224 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.400105    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 20:50:48.404012 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.400696    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:50:48.460471 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.460436    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:50:48.460736 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.460708    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:50:48.487385 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.487348    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:50:48.487717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.487684    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:48.589894 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.568458    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:48.589894 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.569121    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:50:48.730935 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.729990    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:50:48.936184 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:48.917988    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:49.472933 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:49.445840    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 20:50:49.494735 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:49.491310    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:50:49.494735 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:49.491393    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:50:49.527301 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:49.527264    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cwl6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:50:49.527595 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:49.527561    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:49.528834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:49.528805    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 20:50:49.783346 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:49.783301    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:50:49.917792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:49.917754    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"]
Oct 30 20:50:49.947883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:49.944200    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:50:49.992115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:49.992081    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"]
Oct 30 20:50:50.068914 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.068820    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"]
Oct 30 20:50:50.091969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.073022    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:50.130490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.130460    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:50:50.130819 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.130799    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:50:50.231674 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.231632    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd14b38e-xkvtn\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:50.231937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.231915    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd14b38e-xkvtn\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:50.232079 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.232061    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445jz\" (UniqueName: \"kubernetes.io/projected/4eb3ccae-dfa1-48c3-8a52-268a035034ea-kube-api-access-445jz\") pod \"maas-api-revoke-sub-dd14b38e-xkvtn\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:50.269811 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.269769    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q" event={"ID":"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33","Type":"ContainerStarted","Data":"4dcba2ca1051082901543d9d7d42a87741471c3832abfb2ecf1f828a496164a0"}
Oct 30 20:50:50.346935 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.334612    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd14b38e-xkvtn\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:50.370383 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.370345    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd14b38e-xkvtn\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:50.372020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.371990    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445jz\" (UniqueName: \"kubernetes.io/projected/4eb3ccae-dfa1-48c3-8a52-268a035034ea-kube-api-access-445jz\") pod \"maas-api-revoke-sub-dd14b38e-xkvtn\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:50.372682 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.351962    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd14b38e-xkvtn\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:50.429327 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.429292    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd14b38e-xkvtn\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:50.436435 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.436397    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 30 20:50:50.436688 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.436660    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 30 20:50:50.558290 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.553215    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d5dmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:50:50.558290 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.553279    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" podUID="f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:50.558290 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.553628    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445jz\" (UniqueName: \"kubernetes.io/projected/4eb3ccae-dfa1-48c3-8a52-268a035034ea-kube-api-access-445jz\") pod \"maas-api-revoke-sub-dd14b38e-xkvtn\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:50.681356 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:50.681060    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:51.164252 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:51.164219    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"]
Oct 30 20:50:51.226109 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:50:51.225989    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb3ccae_dfa1_48c3_8a52_268a035034ea.slice/crio-7d95c3f2bbff97bd6de3ec61f6952f36a496463cc4c69afd9085fc856d89794e WatchSource:0}: Error finding container 7d95c3f2bbff97bd6de3ec61f6952f36a496463cc4c69afd9085fc856d89794e: Status 404 returned error can't find the container with id 7d95c3f2bbff97bd6de3ec61f6952f36a496463cc4c69afd9085fc856d89794e
Oct 30 20:50:51.299268 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:51.299221    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn" event={"ID":"4eb3ccae-dfa1-48c3-8a52-268a035034ea","Type":"ContainerStarted","Data":"7d95c3f2bbff97bd6de3ec61f6952f36a496463cc4c69afd9085fc856d89794e"}
Oct 30 20:50:51.580211 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:51.580174    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:51.581106 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:51.581033    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:51.582020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:51.581995    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 20:50:52.311709 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:52.311656    4810 generic.go:356] "Generic (PLEG): container finished" podID="4eb3ccae-dfa1-48c3-8a52-268a035034ea" containerID="4da0730e06c09ac3930ce544debcb7360e3088b57eb3dcc5350f09f2bf7da9a6" exitCode=0
Oct 30 20:50:52.312307 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:52.312280    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn" event={"ID":"4eb3ccae-dfa1-48c3-8a52-268a035034ea","Type":"ContainerDied","Data":"4da0730e06c09ac3930ce544debcb7360e3088b57eb3dcc5350f09f2bf7da9a6"}
Oct 30 20:50:53.352224 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.352188    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr" podUID="cb2b8ac5-f431-4fdb-82df-e8f935e561f7" containerName="main" containerID="cri-o://8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d" gracePeriod=45
Oct 30 20:50:53.481079 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.481031    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr" podUID="cb2b8ac5-f431-4fdb-82df-e8f935e561f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 30 20:50:53.527960 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.527932    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:53.603290 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.603257    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-xks-ca\") pod \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") "
Oct 30 20:50:53.603589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.603565    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb3ccae-dfa1-48c3-8a52-268a035034ea-kube-api-access-445jz\" (UniqueName: \"kubernetes.io/projected/4eb3ccae-dfa1-48c3-8a52-268a035034ea-kube-api-access-445jz\") pod \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") "
Oct 30 20:50:53.603657 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.603610    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-openshift-ca\") pod \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\" (UID: \"4eb3ccae-dfa1-48c3-8a52-268a035034ea\") "
Oct 30 20:50:53.604513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.604482    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-openshift-ca" pod "4eb3ccae-dfa1-48c3-8a52-268a035034ea" (UID: "4eb3ccae-dfa1-48c3-8a52-268a035034ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:53.605839 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.605809    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-xks-ca" pod "4eb3ccae-dfa1-48c3-8a52-268a035034ea" (UID: "4eb3ccae-dfa1-48c3-8a52-268a035034ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:50:53.608256 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.608232    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb3ccae-dfa1-48c3-8a52-268a035034ea-kube-api-access-445jz" pod "4eb3ccae-dfa1-48c3-8a52-268a035034ea" (UID: "4eb3ccae-dfa1-48c3-8a52-268a035034ea"). InnerVolumeSpecName "kube-api-access-445jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:53.675889 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.675829    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:53.678638 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.678599    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:50:53.678747 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.678680    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:50:53.704799 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.704764    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tls-certs\") pod \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") "
Oct 30 20:50:53.706104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.705194    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-home\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-home\") pod \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") "
Oct 30 20:50:53.706104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.705243    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-kube-api-access-f6k8c\" (UniqueName: \"kubernetes.io/projected/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-kube-api-access-f6k8c\") pod \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") "
Oct 30 20:50:53.706104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.705307    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-model-cache\") pod \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") "
Oct 30 20:50:53.706104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.705338    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-dshm\") pod \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") "
Oct 30 20:50:53.706104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.705438    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tmp-dir\") pod \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\" (UID: \"cb2b8ac5-f431-4fdb-82df-e8f935e561f7\") "
Oct 30 20:50:53.706104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.705911    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-home" pod "cb2b8ac5-f431-4fdb-82df-e8f935e561f7" (UID: "cb2b8ac5-f431-4fdb-82df-e8f935e561f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:50:53.706104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.706077    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tmp-dir" pod "cb2b8ac5-f431-4fdb-82df-e8f935e561f7" (UID: "cb2b8ac5-f431-4fdb-82df-e8f935e561f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:50:53.706104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.706098    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:53.706460 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.706116    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445jz\" (UniqueName: \"kubernetes.io/projected/4eb3ccae-dfa1-48c3-8a52-268a035034ea-kube-api-access-445jz\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:53.706460 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.706134    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb3ccae-dfa1-48c3-8a52-268a035034ea-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:53.706460 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.706149    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:53.706592 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.706558    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-model-cache" pod "cb2b8ac5-f431-4fdb-82df-e8f935e561f7" (UID: "cb2b8ac5-f431-4fdb-82df-e8f935e561f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:50:53.712232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.709958    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tls-certs" pod "cb2b8ac5-f431-4fdb-82df-e8f935e561f7" (UID: "cb2b8ac5-f431-4fdb-82df-e8f935e561f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:50:53.712232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.711789    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-dshm" pod "cb2b8ac5-f431-4fdb-82df-e8f935e561f7" (UID: "cb2b8ac5-f431-4fdb-82df-e8f935e561f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:50:53.712232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.712059    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-kube-api-access-f6k8c" pod "cb2b8ac5-f431-4fdb-82df-e8f935e561f7" (UID: "cb2b8ac5-f431-4fdb-82df-e8f935e561f7"). InnerVolumeSpecName "kube-api-access-f6k8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:53.807454 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.807412    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:53.807631 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.807488    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6k8c\" (UniqueName: \"kubernetes.io/projected/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-kube-api-access-f6k8c\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:53.807631 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.807508    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:53.807631 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.807524    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:53.807631 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:53.807539    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2b8ac5-f431-4fdb-82df-e8f935e561f7-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:50:54.330846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.330811    4810 generic.go:356] "Generic (PLEG): container finished" podID="cb2b8ac5-f431-4fdb-82df-e8f935e561f7" containerID="8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d" exitCode=0
Oct 30 20:50:54.331057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.330898    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr" event={"ID":"cb2b8ac5-f431-4fdb-82df-e8f935e561f7","Type":"ContainerDied","Data":"8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d"}
Oct 30 20:50:54.331057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.330942    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr" event={"ID":"cb2b8ac5-f431-4fdb-82df-e8f935e561f7","Type":"ContainerDied","Data":"0d998a1a602df21b9256cf8faf961e7c26127c47fc60ef8b06b0331bc1358c2a"}
Oct 30 20:50:54.331057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.330960    4810 scope.go:122] "RemoveContainer" containerID="8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d"
Oct 30 20:50:54.331057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.330908    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"
Oct 30 20:50:54.332921 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.332887    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q" event={"ID":"e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33","Type":"ContainerStarted","Data":"c44b9e037e9c584e59779edd3366419124d349b965299facf4b6d17a1e543dae"}
Oct 30 20:50:54.333061 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.332986    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:54.334981 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.334622    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"
Oct 30 20:50:54.334981 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.334644    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn" event={"ID":"4eb3ccae-dfa1-48c3-8a52-268a035034ea","Type":"ContainerDied","Data":"7d95c3f2bbff97bd6de3ec61f6952f36a496463cc4c69afd9085fc856d89794e"}
Oct 30 20:50:54.334981 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.334682    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d95c3f2bbff97bd6de3ec61f6952f36a496463cc4c69afd9085fc856d89794e"
Oct 30 20:50:54.339227 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.339191    4810 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-k2n4q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:50:54.339318 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.339248    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q" podUID="e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:50:54.343060 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.343039    4810 scope.go:122] "RemoveContainer" containerID="8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d"
Oct 30 20:50:54.343350 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:50:54.343331    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d\": container with ID starting with 8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d not found: ID does not exist" containerID="8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d"
Oct 30 20:50:54.343416 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.343358    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d"} err="failed to get container status \"8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d\": rpc error: code = NotFound desc = could not find container \"8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d\": container with ID starting with 8cf29cf7825b04990779701d6dccd9862f7e7ba956a526f089c58601e8eeb02d not found: ID does not exist"
Oct 30 20:50:54.418740 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.418685    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q" podStartSLOduration=3.400554085 podStartE2EDuration="7.41867087s" totalImagesPullingTime="4.018116785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:54.416393926 +0000 UTC m=+820.908210473" watchObservedRunningTime="2026-10-30 20:50:54.41867087 +0000 UTC m=+820.910487415"
Oct 30 20:50:54.447071 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.447031    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"]
Oct 30 20:50:54.453474 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:54.452282    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/e2e-conflict-b-b39ef5b3-kserve-774f98f86d-rpfxr"]
Oct 30 20:50:55.156697 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:50:55.156658    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:50:55.344003 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:55.343976    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k2n4q"
Oct 30 20:50:55.669697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:55.669664    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb2b8ac5-f431-4fdb-82df-e8f935e561f7" path="/var/lib/kubelet/pods/cb2b8ac5-f431-4fdb-82df-e8f935e561f7/volumes"
Oct 30 20:50:59.825276 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:50:59.825239    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:51:00.262574 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.262542    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"]
Oct 30 20:51:00.263638 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.263611    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4eb3ccae-dfa1-48c3-8a52-268a035034ea" containerName="revoke-keys"
Oct 30 20:51:00.263638 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.263639    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb3ccae-dfa1-48c3-8a52-268a035034ea" containerName="revoke-keys"
Oct 30 20:51:00.263830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.263702    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cb2b8ac5-f431-4fdb-82df-e8f935e561f7" containerName="main"
Oct 30 20:51:00.263830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.263710    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2b8ac5-f431-4fdb-82df-e8f935e561f7" containerName="main"
Oct 30 20:51:00.264007 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.263846    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb3ccae-dfa1-48c3-8a52-268a035034ea" containerName="revoke-keys"
Oct 30 20:51:00.264007 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.263881    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2b8ac5-f431-4fdb-82df-e8f935e561f7" containerName="main"
Oct 30 20:51:00.267451 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.267432    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.273557 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.273084    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:51:00.273557 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.273146    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:51:00.277403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.277378    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"]
Oct 30 20:51:00.382374 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.382342    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczcp\" (UniqueName: \"kubernetes.io/projected/940b3254-9a5c-4465-866c-c41234eea321-kube-api-access-wczcp\") pod \"maas-api-revoke-sub-9591a6da-tn5sp\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.382605 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.382580    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940b3254-9a5c-4465-866c-c41234eea321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9591a6da-tn5sp\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.382733 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.382712    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940b3254-9a5c-4465-866c-c41234eea321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9591a6da-tn5sp\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.484567 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.484526    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940b3254-9a5c-4465-866c-c41234eea321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9591a6da-tn5sp\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.484770 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.484636    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940b3254-9a5c-4465-866c-c41234eea321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9591a6da-tn5sp\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.484770 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.484709    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczcp\" (UniqueName: \"kubernetes.io/projected/940b3254-9a5c-4465-866c-c41234eea321-kube-api-access-wczcp\") pod \"maas-api-revoke-sub-9591a6da-tn5sp\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.485516 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.485483    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940b3254-9a5c-4465-866c-c41234eea321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9591a6da-tn5sp\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.487267 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.487244    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940b3254-9a5c-4465-866c-c41234eea321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9591a6da-tn5sp\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.505654 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.505571    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczcp\" (UniqueName: \"kubernetes.io/projected/940b3254-9a5c-4465-866c-c41234eea321-kube-api-access-wczcp\") pod \"maas-api-revoke-sub-9591a6da-tn5sp\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.592452 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.592358    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:00.972558 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:51:00.972523    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940b3254_9a5c_4465_866c_c41234eea321.slice/crio-2cab6990fd3049bb94a5c5e6fb2942afe08f3c6d3e09b72312d7715ce7b69271 WatchSource:0}: Error finding container 2cab6990fd3049bb94a5c5e6fb2942afe08f3c6d3e09b72312d7715ce7b69271: Status 404 returned error can't find the container with id 2cab6990fd3049bb94a5c5e6fb2942afe08f3c6d3e09b72312d7715ce7b69271
Oct 30 20:51:01.016464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:00.974724    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"]
Oct 30 20:51:01.408059 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:01.408023    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp" event={"ID":"940b3254-9a5c-4465-866c-c41234eea321","Type":"ContainerStarted","Data":"2cab6990fd3049bb94a5c5e6fb2942afe08f3c6d3e09b72312d7715ce7b69271"}
Oct 30 20:51:02.014881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.014557    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"]
Oct 30 20:51:02.019931 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.019902    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.043983 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.043887    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"]
Oct 30 20:51:02.142296 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.142244    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168be12e-db11-4849-99e4-290d38a4e8ed-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.142646 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.142325    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.142646 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.142362    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.142646 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.142390    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168be12e-db11-4849-99e4-290d38a4e8ed-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.142646 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.142432    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.142646 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.142492    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.142646 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.142543    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.142646 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.142573    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.142646 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.142607    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5c72\" (UniqueName: \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-kube-api-access-x5c72\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.243624 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.243587    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.243830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.243640    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.243830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.243663    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.243830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.243692    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5c72\" (UniqueName: \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-kube-api-access-x5c72\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.243830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.243760    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168be12e-db11-4849-99e4-290d38a4e8ed-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.243830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.243815    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.244112 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.243852    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.244112 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.243906    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168be12e-db11-4849-99e4-290d38a4e8ed-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.244112 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.243950    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.244268 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.244160    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.244268 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.244218    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.244361 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.244339    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.244422 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.244402    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.244846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.244822    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168be12e-db11-4849-99e4-290d38a4e8ed-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.246123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.246098    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.246365 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.246346    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168be12e-db11-4849-99e4-290d38a4e8ed-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.253139 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.253118    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.253454 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.253433    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5c72\" (UniqueName: \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-kube-api-access-x5c72\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.349180 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.349097    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:02.425266 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.425236    4810 generic.go:356] "Generic (PLEG): container finished" podID="940b3254-9a5c-4465-866c-c41234eea321" containerID="bb03be2cc9f611cf84e95bbcfe9c2f5025485e92ef36b13b98fc72a805345198" exitCode=0
Oct 30 20:51:02.425408 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.425379    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp" event={"ID":"940b3254-9a5c-4465-866c-c41234eea321","Type":"ContainerDied","Data":"bb03be2cc9f611cf84e95bbcfe9c2f5025485e92ef36b13b98fc72a805345198"}
Oct 30 20:51:02.634065 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.634034    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"]
Oct 30 20:51:02.635509 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:51:02.635484    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168be12e_db11_4849_99e4_290d38a4e8ed.slice/crio-3e9eaa033e4837691afe9050ea906a10137482720a09aed9295b6c4b280174ec WatchSource:0}: Error finding container 3e9eaa033e4837691afe9050ea906a10137482720a09aed9295b6c4b280174ec: Status 404 returned error can't find the container with id 3e9eaa033e4837691afe9050ea906a10137482720a09aed9295b6c4b280174ec
Oct 30 20:51:02.638803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.638766    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:02.638940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.638850    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:02.638940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:02.638917    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:03.433287 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.433246    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" event={"ID":"168be12e-db11-4849-99e4-290d38a4e8ed","Type":"ContainerStarted","Data":"9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538"}
Oct 30 20:51:03.433669 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.433311    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" event={"ID":"168be12e-db11-4849-99e4-290d38a4e8ed","Type":"ContainerStarted","Data":"3e9eaa033e4837691afe9050ea906a10137482720a09aed9295b6c4b280174ec"}
Oct 30 20:51:03.461942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.461852    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" podStartSLOduration=2.461836236 podStartE2EDuration="2.461836236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:03.460087021 +0000 UTC m=+829.951903566" watchObservedRunningTime="2026-10-30 20:51:03.461836236 +0000 UTC m=+829.953652798"
Oct 30 20:51:03.636357 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.636326    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:03.763895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.763809    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940b3254-9a5c-4465-866c-c41234eea321-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940b3254-9a5c-4465-866c-c41234eea321-maas-api-openshift-ca\") pod \"940b3254-9a5c-4465-866c-c41234eea321\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") "
Oct 30 20:51:03.764051 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.763942    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940b3254-9a5c-4465-866c-c41234eea321-kube-api-access-wczcp\" (UniqueName: \"kubernetes.io/projected/940b3254-9a5c-4465-866c-c41234eea321-kube-api-access-wczcp\") pod \"940b3254-9a5c-4465-866c-c41234eea321\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") "
Oct 30 20:51:03.764051 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.763999    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/940b3254-9a5c-4465-866c-c41234eea321-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940b3254-9a5c-4465-866c-c41234eea321-maas-api-xks-ca\") pod \"940b3254-9a5c-4465-866c-c41234eea321\" (UID: \"940b3254-9a5c-4465-866c-c41234eea321\") "
Oct 30 20:51:03.764176 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.764137    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940b3254-9a5c-4465-866c-c41234eea321-maas-api-openshift-ca" pod "940b3254-9a5c-4465-866c-c41234eea321" (UID: "940b3254-9a5c-4465-866c-c41234eea321"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:03.764347 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.764317    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/940b3254-9a5c-4465-866c-c41234eea321-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:03.766294 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.766269    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940b3254-9a5c-4465-866c-c41234eea321-maas-api-xks-ca" pod "940b3254-9a5c-4465-866c-c41234eea321" (UID: "940b3254-9a5c-4465-866c-c41234eea321"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:51:03.766384 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.766311    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940b3254-9a5c-4465-866c-c41234eea321-kube-api-access-wczcp" pod "940b3254-9a5c-4465-866c-c41234eea321" (UID: "940b3254-9a5c-4465-866c-c41234eea321"). InnerVolumeSpecName "kube-api-access-wczcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:03.865708 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.865675    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wczcp\" (UniqueName: \"kubernetes.io/projected/940b3254-9a5c-4465-866c-c41234eea321-kube-api-access-wczcp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:03.865708 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:03.865706    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/940b3254-9a5c-4465-866c-c41234eea321-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:04.349821 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:04.349788    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:04.351578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:04.351554    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:51:04.351715 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:04.351598    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:51:04.441409 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:04.441382    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"
Oct 30 20:51:04.441929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:04.441421    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp" event={"ID":"940b3254-9a5c-4465-866c-c41234eea321","Type":"ContainerDied","Data":"2cab6990fd3049bb94a5c5e6fb2942afe08f3c6d3e09b72312d7715ce7b69271"}
Oct 30 20:51:04.441929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:04.441468    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cab6990fd3049bb94a5c5e6fb2942afe08f3c6d3e09b72312d7715ce7b69271"
Oct 30 20:51:05.349850 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:05.349816    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:51:05.350037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:05.349903    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:51:06.349965 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:06.349924    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:51:06.350369 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:06.349984    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:51:07.210951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.209120    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"]
Oct 30 20:51:07.210951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.209924    4810 cpu_manager.go:414] "containerMap: removing container" podUID="940b3254-9a5c-4465-866c-c41234eea321" containerName="revoke-keys"
Oct 30 20:51:07.210951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.209945    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940b3254-9a5c-4465-866c-c41234eea321" containerName="revoke-keys"
Oct 30 20:51:07.210951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.210157    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="940b3254-9a5c-4465-866c-c41234eea321" containerName="revoke-keys"
Oct 30 20:51:07.215910 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.215664    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"]
Oct 30 20:51:07.215910 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.215778    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.222926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.222902    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:51:07.223338 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.223320    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:51:07.324783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.324740    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75mm\" (UniqueName: \"kubernetes.io/projected/01517e7c-e24a-48a0-bfd3-72a9c5c03135-kube-api-access-m75mm\") pod \"maas-api-revoke-sub-8d9a6002-trskn\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.325002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.324801    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9a6002-trskn\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.325072 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.325039    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9a6002-trskn\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.350151 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.350113    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:51:07.350512 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.350175    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:51:07.425778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.425737    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75mm\" (UniqueName: \"kubernetes.io/projected/01517e7c-e24a-48a0-bfd3-72a9c5c03135-kube-api-access-m75mm\") pod \"maas-api-revoke-sub-8d9a6002-trskn\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.425778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.425782    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9a6002-trskn\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.426034 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.425852    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9a6002-trskn\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.426630 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.426605    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9a6002-trskn\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.428922 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.428896    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9a6002-trskn\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.434359 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.434334    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75mm\" (UniqueName: \"kubernetes.io/projected/01517e7c-e24a-48a0-bfd3-72a9c5c03135-kube-api-access-m75mm\") pod \"maas-api-revoke-sub-8d9a6002-trskn\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.534421 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.534344    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:07.857210 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:07.857182    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"]
Oct 30 20:51:07.859673 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:51:07.859637    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01517e7c_e24a_48a0_bfd3_72a9c5c03135.slice/crio-7b68e885f2189867cc4d5c9abc2cb56eb9115036f38b747ad9ffcafce2ab8a23 WatchSource:0}: Error finding container 7b68e885f2189867cc4d5c9abc2cb56eb9115036f38b747ad9ffcafce2ab8a23: Status 404 returned error can't find the container with id 7b68e885f2189867cc4d5c9abc2cb56eb9115036f38b747ad9ffcafce2ab8a23
Oct 30 20:51:08.349957 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:08.349924    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:51:08.350130 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:08.349974    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:51:08.468066 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:08.468028    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn" event={"ID":"01517e7c-e24a-48a0-bfd3-72a9c5c03135","Type":"ContainerStarted","Data":"1eb87ef27bb22d40dbe909addb1161bb6f392c550cb57bfd997cdb6839fc180f"}
Oct 30 20:51:08.468454 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:08.468091    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn" event={"ID":"01517e7c-e24a-48a0-bfd3-72a9c5c03135","Type":"ContainerStarted","Data":"7b68e885f2189867cc4d5c9abc2cb56eb9115036f38b747ad9ffcafce2ab8a23"}
Oct 30 20:51:09.134711 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.134653    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn" podStartSLOduration=2.134638344 podStartE2EDuration="2.134638344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:08.49619069 +0000 UTC m=+834.988007241" watchObservedRunningTime="2026-10-30 20:51:09.134638344 +0000 UTC m=+835.626454922"
Oct 30 20:51:09.136655 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.136632    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"]
Oct 30 20:51:09.140212 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.140193    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.158185 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.158159    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"]
Oct 30 20:51:09.184066 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.184024    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"]
Oct 30 20:51:09.217376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.217338    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.249515 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.248940    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d300d6-6375-45bb-a33b-1f66ca279494-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.249749 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.249728    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.249929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.249910    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d300d6-6375-45bb-a33b-1f66ca279494-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.250135 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.250116    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.250284 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.250266    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.250484 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.250465    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg8nq\" (UniqueName: \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-kube-api-access-wg8nq\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.252071 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.252045    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.252177 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.252099    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.252177 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.252135    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.253945 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.253875    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"]
Oct 30 20:51:09.350166 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.350126    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:51:09.350330 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.350195    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 20:51:09.353692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.353665    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.353800 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.353702    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.353800 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.353720    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6887c4-2c11-40f4-87f4-1db9dc258170-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.353800 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.353740    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.353967 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.353820    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.353967 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.353942    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d300d6-6375-45bb-a33b-1f66ca279494-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.354087 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.353974    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.354087 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354012    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.354087 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354049    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d300d6-6375-45bb-a33b-1f66ca279494-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.354087 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354079    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.354293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354163    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.354293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354188    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpkfk\" (UniqueName: \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-kube-api-access-bpkfk\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.354293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354204    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.354293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354228    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.354293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354242    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.354293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354266    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.354596 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354437    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.354596 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354496    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.354596 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354519    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.354596 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354551    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.354804 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354618    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg8nq\" (UniqueName: \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-kube-api-access-wg8nq\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.354804 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354666    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.354932 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.354877    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d300d6-6375-45bb-a33b-1f66ca279494-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.356655 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.356631    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d300d6-6375-45bb-a33b-1f66ca279494-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.356808 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.356785    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.362960 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.362936    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.363584 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.363561    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg8nq\" (UniqueName: \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-kube-api-access-wg8nq\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.455485 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.455451    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:09.455888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.455845    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.455964 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.455930    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.456019 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.456002    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6887c4-2c11-40f4-87f4-1db9dc258170-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.456073 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.456034    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.456124 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.456113    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.456180 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.456150    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.456233 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.456219    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.456297 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.456244    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpkfk\" (UniqueName: \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-kube-api-access-bpkfk\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.456297 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.456278    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.456939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.456910    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.457205 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.457182    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.457499 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.457478    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.457729 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.457709    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.457997 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.457977    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6887c4-2c11-40f4-87f4-1db9dc258170-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.458726 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.458697    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.459256 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.459217    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.467822 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.467798    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.468011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.467989    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpkfk\" (UniqueName: \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-kube-api-access-bpkfk\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.483022 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.482987    4810 generic.go:356] "Generic (PLEG): container finished" podID="01517e7c-e24a-48a0-bfd3-72a9c5c03135" containerID="1eb87ef27bb22d40dbe909addb1161bb6f392c550cb57bfd997cdb6839fc180f" exitCode=0
Oct 30 20:51:09.484041 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.483313    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn" event={"ID":"01517e7c-e24a-48a0-bfd3-72a9c5c03135","Type":"ContainerDied","Data":"1eb87ef27bb22d40dbe909addb1161bb6f392c550cb57bfd997cdb6839fc180f"}
Oct 30 20:51:09.569613 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.559355    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:09.829556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.829517    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"]
Oct 30 20:51:09.833605 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:51:09.833563    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d300d6_6375_45bb_a33b_1f66ca279494.slice/crio-44251a6a68701fdabd803b2173251ce0faf0196d0a49f5cdb93900248c184c26 WatchSource:0}: Error finding container 44251a6a68701fdabd803b2173251ce0faf0196d0a49f5cdb93900248c184c26: Status 404 returned error can't find the container with id 44251a6a68701fdabd803b2173251ce0faf0196d0a49f5cdb93900248c184c26
Oct 30 20:51:09.838676 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.837946    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:09.838676 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.838036    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:09.838676 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.838079    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:09.905938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.905825    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"]
Oct 30 20:51:09.908353 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:51:09.908320    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6887c4_2c11_40f4_87f4_1db9dc258170.slice/crio-8c699f7add2b3fabc338b03947f3fad5a68b32f18c445dd878dba33128e92ccd WatchSource:0}: Error finding container 8c699f7add2b3fabc338b03947f3fad5a68b32f18c445dd878dba33128e92ccd: Status 404 returned error can't find the container with id 8c699f7add2b3fabc338b03947f3fad5a68b32f18c445dd878dba33128e92ccd
Oct 30 20:51:09.912681 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.912274    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:09.912681 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.912353    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:09.912681 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:09.912394    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:10.396936 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.394925    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:10.396936 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.395909    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:10.407984 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.407068    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 20:51:10.517881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.517639    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" event={"ID":"34d300d6-6375-45bb-a33b-1f66ca279494","Type":"ContainerStarted","Data":"2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d"}
Oct 30 20:51:10.517881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.517718    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" event={"ID":"34d300d6-6375-45bb-a33b-1f66ca279494","Type":"ContainerStarted","Data":"44251a6a68701fdabd803b2173251ce0faf0196d0a49f5cdb93900248c184c26"}
Oct 30 20:51:10.520777 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.520711    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" event={"ID":"5a6887c4-2c11-40f4-87f4-1db9dc258170","Type":"ContainerStarted","Data":"deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473"}
Oct 30 20:51:10.521008 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.520975    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" event={"ID":"5a6887c4-2c11-40f4-87f4-1db9dc258170","Type":"ContainerStarted","Data":"8c699f7add2b3fabc338b03947f3fad5a68b32f18c445dd878dba33128e92ccd"}
Oct 30 20:51:10.546031 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.544978    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podStartSLOduration=1.54495676 podStartE2EDuration="1.54495676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:10.543100672 +0000 UTC m=+837.034917219" watchObservedRunningTime="2026-10-30 20:51:10.54495676 +0000 UTC m=+837.036773306"
Oct 30 20:51:10.560121 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.560095    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:10.574882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.569038    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" podStartSLOduration=1.569017717 podStartE2EDuration="1.569017717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:10.568776232 +0000 UTC m=+837.060592777" watchObservedRunningTime="2026-10-30 20:51:10.569017717 +0000 UTC m=+837.060834265"
Oct 30 20:51:10.738989 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.738962    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:10.812876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.799658    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-openshift-ca\") pod \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") "
Oct 30 20:51:10.812876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.799730    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-xks-ca\") pod \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") "
Oct 30 20:51:10.812876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.799812    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01517e7c-e24a-48a0-bfd3-72a9c5c03135-kube-api-access-m75mm\" (UniqueName: \"kubernetes.io/projected/01517e7c-e24a-48a0-bfd3-72a9c5c03135-kube-api-access-m75mm\") pod \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\" (UID: \"01517e7c-e24a-48a0-bfd3-72a9c5c03135\") "
Oct 30 20:51:10.812876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.801281    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-openshift-ca" pod "01517e7c-e24a-48a0-bfd3-72a9c5c03135" (UID: "01517e7c-e24a-48a0-bfd3-72a9c5c03135"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:10.837881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.817247    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01517e7c-e24a-48a0-bfd3-72a9c5c03135-kube-api-access-m75mm" pod "01517e7c-e24a-48a0-bfd3-72a9c5c03135" (UID: "01517e7c-e24a-48a0-bfd3-72a9c5c03135"). InnerVolumeSpecName "kube-api-access-m75mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:10.837881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.822970    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-xks-ca" pod "01517e7c-e24a-48a0-bfd3-72a9c5c03135" (UID: "01517e7c-e24a-48a0-bfd3-72a9c5c03135"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:51:10.929882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.923775    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m75mm\" (UniqueName: \"kubernetes.io/projected/01517e7c-e24a-48a0-bfd3-72a9c5c03135-kube-api-access-m75mm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:10.929882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.923814    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:10.929882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:10.923834    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01517e7c-e24a-48a0-bfd3-72a9c5c03135-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:11.456012 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:11.455981    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:11.458274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:11.458179    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:51:11.458274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:11.458225    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:51:11.550413 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:11.550369    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn" event={"ID":"01517e7c-e24a-48a0-bfd3-72a9c5c03135","Type":"ContainerDied","Data":"7b68e885f2189867cc4d5c9abc2cb56eb9115036f38b747ad9ffcafce2ab8a23"}
Oct 30 20:51:11.550875 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:11.550433    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b68e885f2189867cc4d5c9abc2cb56eb9115036f38b747ad9ffcafce2ab8a23"
Oct 30 20:51:11.550875 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:11.550493    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"
Oct 30 20:51:11.561853 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:11.561830    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:51:11.561944 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:11.561902    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:51:12.465075 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:12.465042    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:51:12.465351 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:12.465323    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:51:12.583880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:12.583238    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:51:12.583880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:12.583294    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:51:13.460005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:13.459945    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 20:51:13.507895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:13.502377    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:51:13.507895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:13.502441    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:51:13.564335 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:13.564292    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:51:13.565534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:13.565493    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:51:13.794588 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:13.794493    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:14.464882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:14.463386    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:51:14.464882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:14.463448    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:51:14.562927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:14.562673    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:51:14.562927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:14.562734    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:51:14.796952 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:14.796564    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:14.803057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:14.802989    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:51:15.215881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.205464    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:51:15.469965 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.468371    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:51:15.469965 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.468434    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:51:15.535879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.533643    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"]
Oct 30 20:51:15.535879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.534671    4810 cpu_manager.go:414] "containerMap: removing container" podUID="01517e7c-e24a-48a0-bfd3-72a9c5c03135" containerName="revoke-keys"
Oct 30 20:51:15.535879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.534692    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01517e7c-e24a-48a0-bfd3-72a9c5c03135" containerName="revoke-keys"
Oct 30 20:51:15.535879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.534957    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="01517e7c-e24a-48a0-bfd3-72a9c5c03135" containerName="revoke-keys"
Oct 30 20:51:15.570905 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.564933    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"]
Oct 30 20:51:15.582880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.571243    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:15.610880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.576412    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:51:15.611197 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.611162    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:51:15.616156 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.594316    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:51:15.641146 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.593635    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:51:15.757633 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.757542    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5h26\" (UniqueName: \"kubernetes.io/projected/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-kube-api-access-m5h26\") pod \"maas-api-revoke-sub-f18d81d8-vtdd8\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:15.758068 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.758041    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18d81d8-vtdd8\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:15.758307 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.758288    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18d81d8-vtdd8\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:15.859866 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.859832    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5h26\" (UniqueName: \"kubernetes.io/projected/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-kube-api-access-m5h26\") pod \"maas-api-revoke-sub-f18d81d8-vtdd8\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:15.860313 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.859989    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18d81d8-vtdd8\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:15.860313 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.860030    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18d81d8-vtdd8\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:15.860662 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.860642    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18d81d8-vtdd8\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:15.862438 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.862416    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18d81d8-vtdd8\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:15.868788 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:15.868764    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5h26\" (UniqueName: \"kubernetes.io/projected/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-kube-api-access-m5h26\") pod \"maas-api-revoke-sub-f18d81d8-vtdd8\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:16.070854 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:16.070768    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:16.456805 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:16.456550    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:51:16.456805 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:16.456615    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:51:16.555927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:16.555892    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"]
Oct 30 20:51:16.561085 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:16.560488    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:51:16.561085 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:16.560551    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:51:16.812477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:16.812335    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8" event={"ID":"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e","Type":"ContainerStarted","Data":"697dc3fab10a9414d1e2161d5e0806ba9f9741dcb94765bb05204b3998ab9b4c"}
Oct 30 20:51:17.456179 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:17.456145    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:51:17.456616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:17.456201    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:51:17.560169 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:17.560134    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:51:17.560333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:17.560188    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 20:51:17.823038 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:17.823000    4810 generic.go:356] "Generic (PLEG): container finished" podID="2cb298f5-a1f6-42d5-9a81-bfeb4530d05e" containerID="ba30633cc5ed043733f7c53b42d3d40b7ce79a1f4052f096d6bf8921ca563d67" exitCode=0
Oct 30 20:51:17.823189 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:17.823083    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8" event={"ID":"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e","Type":"ContainerDied","Data":"ba30633cc5ed043733f7c53b42d3d40b7ce79a1f4052f096d6bf8921ca563d67"}
Oct 30 20:51:18.456112 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:18.456074    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:51:18.456305 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:18.456143    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 20:51:19.004116 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.003058    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:19.004282 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.004267    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:19.005394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.005287    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 20:51:19.133898 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.132115    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:19.263841 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.263803    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-kube-api-access-m5h26\" (UniqueName: \"kubernetes.io/projected/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-kube-api-access-m5h26\") pod \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") "
Oct 30 20:51:19.264086 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.264070    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-openshift-ca\") pod \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") "
Oct 30 20:51:19.264172 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.264124    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-xks-ca\") pod \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\" (UID: \"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e\") "
Oct 30 20:51:19.264893 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.264826    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-openshift-ca" pod "2cb298f5-a1f6-42d5-9a81-bfeb4530d05e" (UID: "2cb298f5-a1f6-42d5-9a81-bfeb4530d05e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:19.266325 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.266296    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-xks-ca" pod "2cb298f5-a1f6-42d5-9a81-bfeb4530d05e" (UID: "2cb298f5-a1f6-42d5-9a81-bfeb4530d05e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:51:19.267005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.266619    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-kube-api-access-m5h26" pod "2cb298f5-a1f6-42d5-9a81-bfeb4530d05e" (UID: "2cb298f5-a1f6-42d5-9a81-bfeb4530d05e"). InnerVolumeSpecName "kube-api-access-m5h26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:19.365332 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.365297    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:19.365332 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.365329    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:19.365332 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.365341    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5h26\" (UniqueName: \"kubernetes.io/projected/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e-kube-api-access-m5h26\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:19.498427 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.498344    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"]
Oct 30 20:51:19.499235 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.499209    4810 cpu_manager.go:414] "containerMap: removing container" podUID="2cb298f5-a1f6-42d5-9a81-bfeb4530d05e" containerName="revoke-keys"
Oct 30 20:51:19.499235 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.499234    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb298f5-a1f6-42d5-9a81-bfeb4530d05e" containerName="revoke-keys"
Oct 30 20:51:19.499446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.499421    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb298f5-a1f6-42d5-9a81-bfeb4530d05e" containerName="revoke-keys"
Oct 30 20:51:19.503822 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.503796    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.518978 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.518952    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"]
Oct 30 20:51:19.675249 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.675189    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzgx\" (UniqueName: \"kubernetes.io/projected/b45204b7-d73e-4c52-b047-d3b107f99e25-kube-api-access-6nzgx\") pod \"maas-api-revoke-sub-f8a7d130-b4tnp\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.675987 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.675571    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a7d130-b4tnp\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.675987 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.675742    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a7d130-b4tnp\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.779577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.777352    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a7d130-b4tnp\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.779577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.777507    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a7d130-b4tnp\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.779577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.777642    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzgx\" (UniqueName: \"kubernetes.io/projected/b45204b7-d73e-4c52-b047-d3b107f99e25-kube-api-access-6nzgx\") pod \"maas-api-revoke-sub-f8a7d130-b4tnp\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.779577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.778808    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a7d130-b4tnp\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.782072 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.782045    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a7d130-b4tnp\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.790472 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.790448    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzgx\" (UniqueName: \"kubernetes.io/projected/b45204b7-d73e-4c52-b047-d3b107f99e25-kube-api-access-6nzgx\") pod \"maas-api-revoke-sub-f8a7d130-b4tnp\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.826304 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.826267    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:19.875274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.873874    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"
Oct 30 20:51:19.875274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.874419    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8" event={"ID":"2cb298f5-a1f6-42d5-9a81-bfeb4530d05e","Type":"ContainerDied","Data":"697dc3fab10a9414d1e2161d5e0806ba9f9741dcb94765bb05204b3998ab9b4c"}
Oct 30 20:51:19.875274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:19.874460    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697dc3fab10a9414d1e2161d5e0806ba9f9741dcb94765bb05204b3998ab9b4c"
Oct 30 20:51:20.640406 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:20.601900    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:51:20.640406 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:20.601972    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:20.816263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:20.816217    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"]
Oct 30 20:51:20.888878 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:20.887141    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp" event={"ID":"b45204b7-d73e-4c52-b047-d3b107f99e25","Type":"ContainerStarted","Data":"210fd0c205493913849487ec5a7022ce259540acb9bda31c7bd2b205d85e7941"}
Oct 30 20:51:21.076176 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:21.076147    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:21.076558 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:21.076539    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:21.077355 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:21.077340    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 20:51:21.895011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:21.894979    4810 generic.go:356] "Generic (PLEG): container finished" podID="b45204b7-d73e-4c52-b047-d3b107f99e25" containerID="dd44819e2507df7128eececf934924c440b3e9855d4f15759d05146c2247b407" exitCode=0
Oct 30 20:51:21.895440 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:21.895067    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp" event={"ID":"b45204b7-d73e-4c52-b047-d3b107f99e25","Type":"ContainerDied","Data":"dd44819e2507df7128eececf934924c440b3e9855d4f15759d05146c2247b407"}
Oct 30 20:51:23.051949 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.051926    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:23.184204 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.184121    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-openshift-ca\") pod \"b45204b7-d73e-4c52-b047-d3b107f99e25\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") "
Oct 30 20:51:23.184374 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.184222    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-xks-ca\") pod \"b45204b7-d73e-4c52-b047-d3b107f99e25\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") "
Oct 30 20:51:23.184374 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.184251    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45204b7-d73e-4c52-b047-d3b107f99e25-kube-api-access-6nzgx\" (UniqueName: \"kubernetes.io/projected/b45204b7-d73e-4c52-b047-d3b107f99e25-kube-api-access-6nzgx\") pod \"b45204b7-d73e-4c52-b047-d3b107f99e25\" (UID: \"b45204b7-d73e-4c52-b047-d3b107f99e25\") "
Oct 30 20:51:23.184548 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.184519    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-openshift-ca" pod "b45204b7-d73e-4c52-b047-d3b107f99e25" (UID: "b45204b7-d73e-4c52-b047-d3b107f99e25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:23.186285 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.186255    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-xks-ca" pod "b45204b7-d73e-4c52-b047-d3b107f99e25" (UID: "b45204b7-d73e-4c52-b047-d3b107f99e25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:51:23.186428 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.186404    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45204b7-d73e-4c52-b047-d3b107f99e25-kube-api-access-6nzgx" pod "b45204b7-d73e-4c52-b047-d3b107f99e25" (UID: "b45204b7-d73e-4c52-b047-d3b107f99e25"). InnerVolumeSpecName "kube-api-access-6nzgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:23.285993 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.285962    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:23.285993 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.285995    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nzgx\" (UniqueName: \"kubernetes.io/projected/b45204b7-d73e-4c52-b047-d3b107f99e25-kube-api-access-6nzgx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:23.286148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.286010    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45204b7-d73e-4c52-b047-d3b107f99e25-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:23.910686 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.910655    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"
Oct 30 20:51:23.910840 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.910690    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp" event={"ID":"b45204b7-d73e-4c52-b047-d3b107f99e25","Type":"ContainerDied","Data":"210fd0c205493913849487ec5a7022ce259540acb9bda31c7bd2b205d85e7941"}
Oct 30 20:51:23.910840 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:23.910748    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="210fd0c205493913849487ec5a7022ce259540acb9bda31c7bd2b205d85e7941"
Oct 30 20:51:25.161389 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:51:25.161304    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:51:26.149120 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.149080    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"]
Oct 30 20:51:26.149961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.149930    4810 cpu_manager.go:414] "containerMap: removing container" podUID="b45204b7-d73e-4c52-b047-d3b107f99e25" containerName="revoke-keys"
Oct 30 20:51:26.149961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.149950    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45204b7-d73e-4c52-b047-d3b107f99e25" containerName="revoke-keys"
Oct 30 20:51:26.150115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.150081    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45204b7-d73e-4c52-b047-d3b107f99e25" containerName="revoke-keys"
Oct 30 20:51:26.154014 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.153994    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.169473 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.169099    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"]
Oct 30 20:51:26.203576 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.201127    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"]
Oct 30 20:51:26.203576 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.201190    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"]
Oct 30 20:51:26.203576 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.201292    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.205656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.205629    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:51:26.205835 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.205801    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:51:26.224585 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.224547    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.224851 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.224822    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.225008 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.224916    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ff0f99-80ba-4272-8685-24b8c172e815-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.225008 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.224984    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdwj\" (UniqueName: \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-kube-api-access-vgdwj\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.225275 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.225018    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.225275 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.225051    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.225275 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.225126    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.225275 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.225164    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.225275 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.225196    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.312942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.312906    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:51:26.326195 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326167    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.326341 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326214    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.326341 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326275    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.326341 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326311    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.326509 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326344    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.326509 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326401    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.326509 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326440    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59df\" (UniqueName: \"kubernetes.io/projected/48daa322-4980-4fd8-a2b7-acfe795df6e6-kube-api-access-r59df\") pod \"maas-api-revoke-sub-4e95bf2f-lp5ds\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.326509 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326477    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.326719 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326538    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ff0f99-80ba-4272-8685-24b8c172e815-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.326719 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326561    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e95bf2f-lp5ds\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.326719 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326608    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e95bf2f-lp5ds\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.326719 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326625    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.326719 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.326648    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdwj\" (UniqueName: \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-kube-api-access-vgdwj\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.327278 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.327253    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.327485 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.327464    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.327568 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.327548    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ff0f99-80ba-4272-8685-24b8c172e815-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.327568 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.327504    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.329496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.329473    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.329566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.329489    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.337097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.337069    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.338251 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.338227    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdwj\" (UniqueName: \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-kube-api-access-vgdwj\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.427446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.427374    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59df\" (UniqueName: \"kubernetes.io/projected/48daa322-4980-4fd8-a2b7-acfe795df6e6-kube-api-access-r59df\") pod \"maas-api-revoke-sub-4e95bf2f-lp5ds\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.427575 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.427462    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e95bf2f-lp5ds\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.427575 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.427494    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e95bf2f-lp5ds\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.428323 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.428298    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e95bf2f-lp5ds\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.430260 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.430238    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e95bf2f-lp5ds\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.436414 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.436389    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59df\" (UniqueName: \"kubernetes.io/projected/48daa322-4980-4fd8-a2b7-acfe795df6e6-kube-api-access-r59df\") pod \"maas-api-revoke-sub-4e95bf2f-lp5ds\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.472659 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.472639    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:26.519896 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.519843    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:26.767876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.767844    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"]
Oct 30 20:51:26.769231 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:51:26.769203    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ff0f99_80ba_4272_8685_24b8c172e815.slice/crio-e349bedef9f81acd7e3351513c35be7345531fbeaad5c19ffb42d0cc7d937847 WatchSource:0}: Error finding container e349bedef9f81acd7e3351513c35be7345531fbeaad5c19ffb42d0cc7d937847: Status 404 returned error can't find the container with id e349bedef9f81acd7e3351513c35be7345531fbeaad5c19ffb42d0cc7d937847
Oct 30 20:51:26.771467 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.771434    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:26.771546 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.771499    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:26.771546 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.771533    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:51:26.837222 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.837197    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"]
Oct 30 20:51:26.838851 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:51:26.838824    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48daa322_4980_4fd8_a2b7_acfe795df6e6.slice/crio-1cdbc7f7d39434a05ae41023dd6b6ef1bdc26dff8aae8c9657e51f0c54fdaea3 WatchSource:0}: Error finding container 1cdbc7f7d39434a05ae41023dd6b6ef1bdc26dff8aae8c9657e51f0c54fdaea3: Status 404 returned error can't find the container with id 1cdbc7f7d39434a05ae41023dd6b6ef1bdc26dff8aae8c9657e51f0c54fdaea3
Oct 30 20:51:26.938253 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.938214    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" event={"ID":"a8ff0f99-80ba-4272-8685-24b8c172e815","Type":"ContainerStarted","Data":"0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9"}
Oct 30 20:51:26.938438 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.938286    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" event={"ID":"a8ff0f99-80ba-4272-8685-24b8c172e815","Type":"ContainerStarted","Data":"e349bedef9f81acd7e3351513c35be7345531fbeaad5c19ffb42d0cc7d937847"}
Oct 30 20:51:26.940352 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.940325    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds" event={"ID":"48daa322-4980-4fd8-a2b7-acfe795df6e6","Type":"ContainerStarted","Data":"1cdbc7f7d39434a05ae41023dd6b6ef1bdc26dff8aae8c9657e51f0c54fdaea3"}
Oct 30 20:51:26.974058 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:26.974000    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podStartSLOduration=0.973980769 podStartE2EDuration="973.980769ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:26.969721434 +0000 UTC m=+853.461537979" watchObservedRunningTime="2026-10-30 20:51:26.973980769 +0000 UTC m=+853.465797315"
Oct 30 20:51:27.473722 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:27.473643    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:27.479342 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:27.479005    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:51:27.479342 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:27.479060    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:51:27.997975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:27.997939    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds" event={"ID":"48daa322-4980-4fd8-a2b7-acfe795df6e6","Type":"ContainerStarted","Data":"cf90798f8ed97e8e89eb6529c67ccb95587c04b6cc2e8708c2e06c72bfbd3594"}
Oct 30 20:51:28.234937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:28.226196    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds" podStartSLOduration=2.226172077 podStartE2EDuration="2.226172077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:28.173898985 +0000 UTC m=+854.665715532" watchObservedRunningTime="2026-10-30 20:51:28.226172077 +0000 UTC m=+854.717988623"
Oct 30 20:51:28.473362 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:28.473325    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:51:28.473643 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:28.473599    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:51:29.020881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:29.019027    4810 generic.go:356] "Generic (PLEG): container finished" podID="48daa322-4980-4fd8-a2b7-acfe795df6e6" containerID="cf90798f8ed97e8e89eb6529c67ccb95587c04b6cc2e8708c2e06c72bfbd3594" exitCode=0
Oct 30 20:51:29.020881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:29.020291    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds" event={"ID":"48daa322-4980-4fd8-a2b7-acfe795df6e6","Type":"ContainerDied","Data":"cf90798f8ed97e8e89eb6529c67ccb95587c04b6cc2e8708c2e06c72bfbd3594"}
Oct 30 20:51:29.231881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:29.231024    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 20:51:29.481881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:29.481155    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:51:29.481881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:29.481230    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:51:30.298883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:30.285957    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:30.311876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:30.303219    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:51:30.352321 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:30.352277    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:51:30.438168 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:30.436944    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:30.456141 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:30.454979    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:30.473611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:30.473214    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:51:30.473611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:30.473269    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:51:30.630896 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:30.554451    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 20:51:31.318883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.314952    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:31.367324 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.365935    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:31.387889 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.368054    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:51:31.562885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.479976    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:31.562885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.506459    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:51:31.562885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.508044    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:31.562885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.512467    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:51:31.562885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.558007    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:31.562885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.561153    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:51:31.562885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.561977    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:31.590935 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.566962    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:51:31.590935 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.567012    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:51:31.730758 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:31.730714    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:32.140350 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.140280    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:51:32.287883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.224419    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:32.461882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.455927    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:51:32.518957 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.518913    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 20:51:32.609886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.513282    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:32.609886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.534021    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:51:32.609886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.607325    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:51:32.609886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.595964    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:32.619065 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.618144    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-xks-ca\") pod \"48daa322-4980-4fd8-a2b7-acfe795df6e6\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") "
Oct 30 20:51:32.619065 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.618217    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48daa322-4980-4fd8-a2b7-acfe795df6e6-kube-api-access-r59df\" (UniqueName: \"kubernetes.io/projected/48daa322-4980-4fd8-a2b7-acfe795df6e6-kube-api-access-r59df\") pod \"48daa322-4980-4fd8-a2b7-acfe795df6e6\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") "
Oct 30 20:51:32.619065 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.618289    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-openshift-ca\") pod \"48daa322-4980-4fd8-a2b7-acfe795df6e6\" (UID: \"48daa322-4980-4fd8-a2b7-acfe795df6e6\") "
Oct 30 20:51:32.619792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.619489    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-openshift-ca" pod "48daa322-4980-4fd8-a2b7-acfe795df6e6" (UID: "48daa322-4980-4fd8-a2b7-acfe795df6e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:32.702511 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.661642    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f8lmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:51:32.702511 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.661712    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f8lmw" podUID="9e300ef6-e571-4178-9b9b-b7a558122275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:32.730823 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.730788    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:32.755468 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.755427    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds" event={"ID":"48daa322-4980-4fd8-a2b7-acfe795df6e6","Type":"ContainerDied","Data":"1cdbc7f7d39434a05ae41023dd6b6ef1bdc26dff8aae8c9657e51f0c54fdaea3"}
Oct 30 20:51:32.755748 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.755725    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cdbc7f7d39434a05ae41023dd6b6ef1bdc26dff8aae8c9657e51f0c54fdaea3"
Oct 30 20:51:32.755956 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.755940    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"
Oct 30 20:51:32.765174 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.762042    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-xks-ca" pod "48daa322-4980-4fd8-a2b7-acfe795df6e6" (UID: "48daa322-4980-4fd8-a2b7-acfe795df6e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:51:32.840581 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.840540    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48daa322-4980-4fd8-a2b7-acfe795df6e6-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:32.896625 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:32.896564    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48daa322-4980-4fd8-a2b7-acfe795df6e6-kube-api-access-r59df" pod "48daa322-4980-4fd8-a2b7-acfe795df6e6" (UID: "48daa322-4980-4fd8-a2b7-acfe795df6e6"). InnerVolumeSpecName "kube-api-access-r59df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:33.000262 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:33.000139    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59df\" (UniqueName: \"kubernetes.io/projected/48daa322-4980-4fd8-a2b7-acfe795df6e6-kube-api-access-r59df\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:33.162884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:33.140945    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:33.464693 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:33.464065    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:33.483190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:33.482740    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:51:33.483190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:33.482798    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:51:33.506151 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:33.505971    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:51:33.506151 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:33.506027    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:33.736886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:33.716077    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 20:51:33.910882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:33.888088    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:34.062252 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:34.062225    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:51:34.078882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:34.071064    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:51:34.078882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:34.071146    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:51:34.078882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:34.071204    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:51:34.159018 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:34.151831    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:34.159018 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:34.154318    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:51:34.237487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:34.237405    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:51:34.483000 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:34.482965    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:51:34.506461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:34.506385    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:51:35.475882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:35.475219    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:51:35.475882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:35.475277    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:51:36.492260 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:36.492222    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:51:36.492846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:36.492814    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:51:38.517616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:38.517575    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:51:38.518314 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:38.518268    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:39.081815 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:39.081779    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:39.082346 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:39.082325    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:39.083148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:39.083127    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 20:51:40.577924 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.577889    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"]
Oct 30 20:51:40.578771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.578752    4810 cpu_manager.go:414] "containerMap: removing container" podUID="48daa322-4980-4fd8-a2b7-acfe795df6e6" containerName="revoke-keys"
Oct 30 20:51:40.578833 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.578773    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48daa322-4980-4fd8-a2b7-acfe795df6e6" containerName="revoke-keys"
Oct 30 20:51:40.578950 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.578938    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="48daa322-4980-4fd8-a2b7-acfe795df6e6" containerName="revoke-keys"
Oct 30 20:51:40.583665 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.583641    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:40.589209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.589181    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:51:40.589352 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.589262    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:51:40.592427 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.592400    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"]
Oct 30 20:51:40.668278 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.668248    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d85271a-6dbkq\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:40.668440 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.668293    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829jf\" (UniqueName: \"kubernetes.io/projected/3a92702f-9cea-4251-a2d2-b873bf2898d7-kube-api-access-829jf\") pod \"maas-api-revoke-sub-1d85271a-6dbkq\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:40.668440 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.668397    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d85271a-6dbkq\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:40.769342 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.769305    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d85271a-6dbkq\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:40.769693 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.769670    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829jf\" (UniqueName: \"kubernetes.io/projected/3a92702f-9cea-4251-a2d2-b873bf2898d7-kube-api-access-829jf\") pod \"maas-api-revoke-sub-1d85271a-6dbkq\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:40.769922 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.769904    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d85271a-6dbkq\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:40.770119 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.770097    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d85271a-6dbkq\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:40.772634 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.772607    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d85271a-6dbkq\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:40.778445 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.778421    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-829jf\" (UniqueName: \"kubernetes.io/projected/3a92702f-9cea-4251-a2d2-b873bf2898d7-kube-api-access-829jf\") pod \"maas-api-revoke-sub-1d85271a-6dbkq\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:40.902081 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:40.902047    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:41.192136 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:41.192095    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"]
Oct 30 20:51:41.193511 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:51:41.193483    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a92702f_9cea_4251_a2d2_b873bf2898d7.slice/crio-fefa3d307df36f5a4af423bb516736c2b5e0cef20d8c4043ed1cfa2ef6e28f39 WatchSource:0}: Error finding container fefa3d307df36f5a4af423bb516736c2b5e0cef20d8c4043ed1cfa2ef6e28f39: Status 404 returned error can't find the container with id fefa3d307df36f5a4af423bb516736c2b5e0cef20d8c4043ed1cfa2ef6e28f39
Oct 30 20:51:42.163757 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:42.163716    4810 generic.go:356] "Generic (PLEG): container finished" podID="3a92702f-9cea-4251-a2d2-b873bf2898d7" containerID="059ba92d7401140d973d7bb894a0096094cc9a2de4b2eb82382009d1f75ce689" exitCode=0
Oct 30 20:51:42.164177 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:42.163793    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq" event={"ID":"3a92702f-9cea-4251-a2d2-b873bf2898d7","Type":"ContainerDied","Data":"059ba92d7401140d973d7bb894a0096094cc9a2de4b2eb82382009d1f75ce689"}
Oct 30 20:51:42.164177 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:42.163886    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq" event={"ID":"3a92702f-9cea-4251-a2d2-b873bf2898d7","Type":"ContainerStarted","Data":"fefa3d307df36f5a4af423bb516736c2b5e0cef20d8c4043ed1cfa2ef6e28f39"}
Oct 30 20:51:43.373377 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:43.373354    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:43.506335 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:43.506249    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-openshift-ca\") pod \"3a92702f-9cea-4251-a2d2-b873bf2898d7\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") "
Oct 30 20:51:43.506513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:43.506379    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-xks-ca\") pod \"3a92702f-9cea-4251-a2d2-b873bf2898d7\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") "
Oct 30 20:51:43.506513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:43.506473    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a92702f-9cea-4251-a2d2-b873bf2898d7-kube-api-access-829jf\" (UniqueName: \"kubernetes.io/projected/3a92702f-9cea-4251-a2d2-b873bf2898d7-kube-api-access-829jf\") pod \"3a92702f-9cea-4251-a2d2-b873bf2898d7\" (UID: \"3a92702f-9cea-4251-a2d2-b873bf2898d7\") "
Oct 30 20:51:43.506675 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:43.506637    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-openshift-ca" pod "3a92702f-9cea-4251-a2d2-b873bf2898d7" (UID: "3a92702f-9cea-4251-a2d2-b873bf2898d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:43.507032 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:43.507011    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:43.508663 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:43.508632    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-xks-ca" pod "3a92702f-9cea-4251-a2d2-b873bf2898d7" (UID: "3a92702f-9cea-4251-a2d2-b873bf2898d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:51:43.508663 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:43.508656    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a92702f-9cea-4251-a2d2-b873bf2898d7-kube-api-access-829jf" pod "3a92702f-9cea-4251-a2d2-b873bf2898d7" (UID: "3a92702f-9cea-4251-a2d2-b873bf2898d7"). InnerVolumeSpecName "kube-api-access-829jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:43.608891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:43.608719    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-829jf\" (UniqueName: \"kubernetes.io/projected/3a92702f-9cea-4251-a2d2-b873bf2898d7-kube-api-access-829jf\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:43.608891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:43.608751    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a92702f-9cea-4251-a2d2-b873bf2898d7-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:44.182997 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:44.182953    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq" event={"ID":"3a92702f-9cea-4251-a2d2-b873bf2898d7","Type":"ContainerDied","Data":"fefa3d307df36f5a4af423bb516736c2b5e0cef20d8c4043ed1cfa2ef6e28f39"}
Oct 30 20:51:44.183246 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:44.183013    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"
Oct 30 20:51:44.183246 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:44.183049    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fefa3d307df36f5a4af423bb516736c2b5e0cef20d8c4043ed1cfa2ef6e28f39"
Oct 30 20:51:46.611881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:46.610936    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 20:51:46.611881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:46.611373    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 20:51:46.661158 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:46.661114    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:47.253885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:47.251963    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:47.612035 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:47.611934    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 20:51:47.612926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:47.612904    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:51:47.655301 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:47.655262    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:47.669096 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:47.669053    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:47.773516 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:47.773465    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:48.103486 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.103452    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:51:48.104146 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.104126    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:51:48.243758 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.243643    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:51:48.257880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.252219    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:48.257880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.256895    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:51:48.409399 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.408934    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:51:48.409399 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.409011    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:48.506379 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.491375    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:51:48.523879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.515283    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 20:51:48.620886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.617905    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:48.975929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:48.974164    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:51:49.260378 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:49.260266    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cwl6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:51:49.260647 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:49.260600    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:51:50.811385 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.811352    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"]
Oct 30 20:51:50.811941 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.811925    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3a92702f-9cea-4251-a2d2-b873bf2898d7" containerName="revoke-keys"
Oct 30 20:51:50.811941 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.811941    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a92702f-9cea-4251-a2d2-b873bf2898d7" containerName="revoke-keys"
Oct 30 20:51:50.812072 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.812062    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a92702f-9cea-4251-a2d2-b873bf2898d7" containerName="revoke-keys"
Oct 30 20:51:50.815798 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.815778    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:50.821096 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.821074    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:51:50.821424 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.821402    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:51:50.824787 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.824765    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"]
Oct 30 20:51:50.943612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.943269    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95edb6c8-hfqx5\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:50.943612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.943411    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwshc\" (UniqueName: \"kubernetes.io/projected/ab448cbe-adc2-4113-a5ed-0dc76a012875-kube-api-access-wwshc\") pod \"maas-api-revoke-sub-95edb6c8-hfqx5\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:50.943612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:50.943491    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95edb6c8-hfqx5\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:51.044730 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:51.044692    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95edb6c8-hfqx5\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:51.044926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:51.044784    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95edb6c8-hfqx5\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:51.044926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:51.044889    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwshc\" (UniqueName: \"kubernetes.io/projected/ab448cbe-adc2-4113-a5ed-0dc76a012875-kube-api-access-wwshc\") pod \"maas-api-revoke-sub-95edb6c8-hfqx5\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:51.045477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:51.045454    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95edb6c8-hfqx5\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:51.047194 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:51.047172    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95edb6c8-hfqx5\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:51.055943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:51.055923    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwshc\" (UniqueName: \"kubernetes.io/projected/ab448cbe-adc2-4113-a5ed-0dc76a012875-kube-api-access-wwshc\") pod \"maas-api-revoke-sub-95edb6c8-hfqx5\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:51.133566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:51.133534    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:51.419306 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:51:51.419274    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab448cbe_adc2_4113_a5ed_0dc76a012875.slice/crio-282e0109237d94b75ced03eecc15c2c41164df92b42dd5dc61196796db50734e WatchSource:0}: Error finding container 282e0109237d94b75ced03eecc15c2c41164df92b42dd5dc61196796db50734e: Status 404 returned error can't find the container with id 282e0109237d94b75ced03eecc15c2c41164df92b42dd5dc61196796db50734e
Oct 30 20:51:51.419523 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:51.419498    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"]
Oct 30 20:51:51.479211 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:51.479188    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5" event={"ID":"ab448cbe-adc2-4113-a5ed-0dc76a012875","Type":"ContainerStarted","Data":"282e0109237d94b75ced03eecc15c2c41164df92b42dd5dc61196796db50734e"}
Oct 30 20:51:52.488876 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:52.488822    4810 generic.go:356] "Generic (PLEG): container finished" podID="ab448cbe-adc2-4113-a5ed-0dc76a012875" containerID="9b5c3a42aa9c3975f599ecdf1be8c81a943e220cacc4d308710ea1992a7d784b" exitCode=0
Oct 30 20:51:52.489260 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:52.488908    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5" event={"ID":"ab448cbe-adc2-4113-a5ed-0dc76a012875","Type":"ContainerDied","Data":"9b5c3a42aa9c3975f599ecdf1be8c81a943e220cacc4d308710ea1992a7d784b"}
Oct 30 20:51:53.666647 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:53.666625    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:53.774176 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:53.774096    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-xks-ca\") pod \"ab448cbe-adc2-4113-a5ed-0dc76a012875\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") "
Oct 30 20:51:53.774341 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:53.774208    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-openshift-ca\") pod \"ab448cbe-adc2-4113-a5ed-0dc76a012875\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") "
Oct 30 20:51:53.774403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:53.774378    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab448cbe-adc2-4113-a5ed-0dc76a012875-kube-api-access-wwshc\" (UniqueName: \"kubernetes.io/projected/ab448cbe-adc2-4113-a5ed-0dc76a012875-kube-api-access-wwshc\") pod \"ab448cbe-adc2-4113-a5ed-0dc76a012875\" (UID: \"ab448cbe-adc2-4113-a5ed-0dc76a012875\") "
Oct 30 20:51:53.774619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:53.774588    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-openshift-ca" pod "ab448cbe-adc2-4113-a5ed-0dc76a012875" (UID: "ab448cbe-adc2-4113-a5ed-0dc76a012875"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:53.774785 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:53.774766    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:53.776618 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:53.776586    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab448cbe-adc2-4113-a5ed-0dc76a012875-kube-api-access-wwshc" pod "ab448cbe-adc2-4113-a5ed-0dc76a012875" (UID: "ab448cbe-adc2-4113-a5ed-0dc76a012875"). InnerVolumeSpecName "kube-api-access-wwshc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:53.777315 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:53.777294    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-xks-ca" pod "ab448cbe-adc2-4113-a5ed-0dc76a012875" (UID: "ab448cbe-adc2-4113-a5ed-0dc76a012875"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:51:53.876163 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:53.876129    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwshc\" (UniqueName: \"kubernetes.io/projected/ab448cbe-adc2-4113-a5ed-0dc76a012875-kube-api-access-wwshc\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:53.876163 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:53.876161    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab448cbe-adc2-4113-a5ed-0dc76a012875-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:51:54.507092 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:54.507059    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"
Oct 30 20:51:54.507262 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:54.507069    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5" event={"ID":"ab448cbe-adc2-4113-a5ed-0dc76a012875","Type":"ContainerDied","Data":"282e0109237d94b75ced03eecc15c2c41164df92b42dd5dc61196796db50734e"}
Oct 30 20:51:54.507262 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:54.507186    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="282e0109237d94b75ced03eecc15c2c41164df92b42dd5dc61196796db50734e"
Oct 30 20:51:54.532828 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:51:54.532789    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:51:55.156963 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:51:55.156848    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:52:00.574445 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.574417    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"]
Oct 30 20:52:00.574978 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.574963    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ab448cbe-adc2-4113-a5ed-0dc76a012875" containerName="revoke-keys"
Oct 30 20:52:00.575034 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.574979    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab448cbe-adc2-4113-a5ed-0dc76a012875" containerName="revoke-keys"
Oct 30 20:52:00.575106 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.575095    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab448cbe-adc2-4113-a5ed-0dc76a012875" containerName="revoke-keys"
Oct 30 20:52:00.578465 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.578448    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:00.584099 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.584077    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:52:00.584343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.584321    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:52:00.591281 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.588291    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"]
Oct 30 20:52:00.661155 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.661126    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e019fd-8gjb6\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:00.661301 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.661161    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e019fd-8gjb6\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:00.661301 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.661293    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrnw\" (UniqueName: \"kubernetes.io/projected/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-kube-api-access-xbrnw\") pod \"maas-api-revoke-sub-e4e019fd-8gjb6\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:00.762593 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.762564    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e019fd-8gjb6\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:00.762771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.762650    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrnw\" (UniqueName: \"kubernetes.io/projected/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-kube-api-access-xbrnw\") pod \"maas-api-revoke-sub-e4e019fd-8gjb6\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:00.762771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.762759    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e019fd-8gjb6\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:00.763334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.763314    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e019fd-8gjb6\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:00.765031 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.765011    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e019fd-8gjb6\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:00.772093 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.772066    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrnw\" (UniqueName: \"kubernetes.io/projected/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-kube-api-access-xbrnw\") pod \"maas-api-revoke-sub-e4e019fd-8gjb6\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:00.897849 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:00.897810    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:01.180708 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:01.180686    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"]
Oct 30 20:52:01.182874 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:52:01.182828    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebc6da4_0e45_4670_92c0_015c1d5e3b17.slice/crio-1a42009a4f25ca6d0afd4aec299e2ad0b3ec445176cecaae0d5ceda5fa2bb626 WatchSource:0}: Error finding container 1a42009a4f25ca6d0afd4aec299e2ad0b3ec445176cecaae0d5ceda5fa2bb626: Status 404 returned error can't find the container with id 1a42009a4f25ca6d0afd4aec299e2ad0b3ec445176cecaae0d5ceda5fa2bb626
Oct 30 20:52:01.561343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:01.561307    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6" event={"ID":"7ebc6da4-0e45-4670-92c0-015c1d5e3b17","Type":"ContainerStarted","Data":"0fbd85dad403bc3a6270bb0e712f502e6fac691272c49998ee0a1604f2e66400"}
Oct 30 20:52:01.561528 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:01.561375    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6" event={"ID":"7ebc6da4-0e45-4670-92c0-015c1d5e3b17","Type":"ContainerStarted","Data":"1a42009a4f25ca6d0afd4aec299e2ad0b3ec445176cecaae0d5ceda5fa2bb626"}
Oct 30 20:52:01.580878 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:01.580797    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6" podStartSLOduration=1.580778017 podStartE2EDuration="1.580778017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:01.5768417 +0000 UTC m=+888.068658245" watchObservedRunningTime="2026-10-30 20:52:01.580778017 +0000 UTC m=+888.072594563"
Oct 30 20:52:02.569925 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:02.569891    4810 generic.go:356] "Generic (PLEG): container finished" podID="7ebc6da4-0e45-4670-92c0-015c1d5e3b17" containerID="0fbd85dad403bc3a6270bb0e712f502e6fac691272c49998ee0a1604f2e66400" exitCode=0
Oct 30 20:52:02.570096 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:02.569950    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6" event={"ID":"7ebc6da4-0e45-4670-92c0-015c1d5e3b17","Type":"ContainerDied","Data":"0fbd85dad403bc3a6270bb0e712f502e6fac691272c49998ee0a1604f2e66400"}
Oct 30 20:52:03.862920 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:03.862896    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:03.905740 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:03.905704    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-openshift-ca\") pod \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") "
Oct 30 20:52:03.906157 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:03.905790    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-kube-api-access-xbrnw\" (UniqueName: \"kubernetes.io/projected/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-kube-api-access-xbrnw\") pod \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") "
Oct 30 20:52:03.906157 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:03.905828    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-xks-ca\") pod \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\" (UID: \"7ebc6da4-0e45-4670-92c0-015c1d5e3b17\") "
Oct 30 20:52:03.906157 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:03.906097    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-openshift-ca" pod "7ebc6da4-0e45-4670-92c0-015c1d5e3b17" (UID: "7ebc6da4-0e45-4670-92c0-015c1d5e3b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:03.906403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:03.906374    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:03.918197 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:03.917988    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-xks-ca" pod "7ebc6da4-0e45-4670-92c0-015c1d5e3b17" (UID: "7ebc6da4-0e45-4670-92c0-015c1d5e3b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:03.918197 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:03.918007    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-kube-api-access-xbrnw" pod "7ebc6da4-0e45-4670-92c0-015c1d5e3b17" (UID: "7ebc6da4-0e45-4670-92c0-015c1d5e3b17"). InnerVolumeSpecName "kube-api-access-xbrnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:04.007405 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.007373    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbrnw\" (UniqueName: \"kubernetes.io/projected/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-kube-api-access-xbrnw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:04.007405 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.007403    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ebc6da4-0e45-4670-92c0-015c1d5e3b17-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:04.588312 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.588272    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"
Oct 30 20:52:04.588312 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.588295    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6" event={"ID":"7ebc6da4-0e45-4670-92c0-015c1d5e3b17","Type":"ContainerDied","Data":"1a42009a4f25ca6d0afd4aec299e2ad0b3ec445176cecaae0d5ceda5fa2bb626"}
Oct 30 20:52:04.588574 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.588345    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a42009a4f25ca6d0afd4aec299e2ad0b3ec445176cecaae0d5ceda5fa2bb626"
Oct 30 20:52:04.719551 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.719517    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"]
Oct 30 20:52:04.720170 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.720150    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7ebc6da4-0e45-4670-92c0-015c1d5e3b17" containerName="revoke-keys"
Oct 30 20:52:04.720170 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.720169    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebc6da4-0e45-4670-92c0-015c1d5e3b17" containerName="revoke-keys"
Oct 30 20:52:04.720329 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.720317    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebc6da4-0e45-4670-92c0-015c1d5e3b17" containerName="revoke-keys"
Oct 30 20:52:04.724006 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.723987    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:04.728664 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.728628    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:52:04.728999 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.728981    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:52:04.733577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.733411    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"]
Oct 30 20:52:04.818446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.818411    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb27p\" (UniqueName: \"kubernetes.io/projected/27e0a0a8-ddd7-4d22-91cf-e11612132492-kube-api-access-fb27p\") pod \"maas-api-revoke-sub-bd796643-rx24v\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:04.818611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.818463    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd796643-rx24v\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:04.818611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.818558    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd796643-rx24v\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:04.919490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.919457    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb27p\" (UniqueName: \"kubernetes.io/projected/27e0a0a8-ddd7-4d22-91cf-e11612132492-kube-api-access-fb27p\") pod \"maas-api-revoke-sub-bd796643-rx24v\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:04.919926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.919502    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd796643-rx24v\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:04.919926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.919533    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd796643-rx24v\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:04.920384 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.920361    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd796643-rx24v\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:04.921999 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.921974    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd796643-rx24v\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:04.930331 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:04.930305    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb27p\" (UniqueName: \"kubernetes.io/projected/27e0a0a8-ddd7-4d22-91cf-e11612132492-kube-api-access-fb27p\") pod \"maas-api-revoke-sub-bd796643-rx24v\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:05.050110 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:05.049835    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:05.349538 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:52:05.349512    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e0a0a8_ddd7_4d22_91cf_e11612132492.slice/crio-ee0fdbb5228991fe00f8c37f7a9a2518a7e33b3a3da998f367ac4e9498fe8698 WatchSource:0}: Error finding container ee0fdbb5228991fe00f8c37f7a9a2518a7e33b3a3da998f367ac4e9498fe8698: Status 404 returned error can't find the container with id ee0fdbb5228991fe00f8c37f7a9a2518a7e33b3a3da998f367ac4e9498fe8698
Oct 30 20:52:05.350011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:05.349987    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"]
Oct 30 20:52:05.600517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:05.600437    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v" event={"ID":"27e0a0a8-ddd7-4d22-91cf-e11612132492","Type":"ContainerStarted","Data":"ee0fdbb5228991fe00f8c37f7a9a2518a7e33b3a3da998f367ac4e9498fe8698"}
Oct 30 20:52:06.608804 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:06.608718    4810 generic.go:356] "Generic (PLEG): container finished" podID="27e0a0a8-ddd7-4d22-91cf-e11612132492" containerID="e9c0d04c023eaeabb38487abbfd6b29b72ea73ff5e1d8dad09d39626c1769722" exitCode=0
Oct 30 20:52:06.609195 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:06.608801    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v" event={"ID":"27e0a0a8-ddd7-4d22-91cf-e11612132492","Type":"ContainerDied","Data":"e9c0d04c023eaeabb38487abbfd6b29b72ea73ff5e1d8dad09d39626c1769722"}
Oct 30 20:52:07.769022 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:07.768997    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:07.860678 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:07.860649    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-openshift-ca\") pod \"27e0a0a8-ddd7-4d22-91cf-e11612132492\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") "
Oct 30 20:52:07.860838 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:07.860743    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-xks-ca\") pod \"27e0a0a8-ddd7-4d22-91cf-e11612132492\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") "
Oct 30 20:52:07.860838 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:07.860777    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e0a0a8-ddd7-4d22-91cf-e11612132492-kube-api-access-fb27p\" (UniqueName: \"kubernetes.io/projected/27e0a0a8-ddd7-4d22-91cf-e11612132492-kube-api-access-fb27p\") pod \"27e0a0a8-ddd7-4d22-91cf-e11612132492\" (UID: \"27e0a0a8-ddd7-4d22-91cf-e11612132492\") "
Oct 30 20:52:07.861079 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:07.861046    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-openshift-ca" pod "27e0a0a8-ddd7-4d22-91cf-e11612132492" (UID: "27e0a0a8-ddd7-4d22-91cf-e11612132492"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:07.862905 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:07.862882    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e0a0a8-ddd7-4d22-91cf-e11612132492-kube-api-access-fb27p" pod "27e0a0a8-ddd7-4d22-91cf-e11612132492" (UID: "27e0a0a8-ddd7-4d22-91cf-e11612132492"). InnerVolumeSpecName "kube-api-access-fb27p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:07.862992 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:07.862945    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-xks-ca" pod "27e0a0a8-ddd7-4d22-91cf-e11612132492" (UID: "27e0a0a8-ddd7-4d22-91cf-e11612132492"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:07.962528 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:07.962501    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:07.962528 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:07.962526    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb27p\" (UniqueName: \"kubernetes.io/projected/27e0a0a8-ddd7-4d22-91cf-e11612132492-kube-api-access-fb27p\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:07.962528 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:07.962537    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e0a0a8-ddd7-4d22-91cf-e11612132492-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:08.627105 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:08.627066    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v" event={"ID":"27e0a0a8-ddd7-4d22-91cf-e11612132492","Type":"ContainerDied","Data":"ee0fdbb5228991fe00f8c37f7a9a2518a7e33b3a3da998f367ac4e9498fe8698"}
Oct 30 20:52:08.627247 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:08.627134    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee0fdbb5228991fe00f8c37f7a9a2518a7e33b3a3da998f367ac4e9498fe8698"
Oct 30 20:52:08.627247 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:08.627077    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"
Oct 30 20:52:10.838582 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:10.838549    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"]
Oct 30 20:52:10.839132 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:10.839118    4810 cpu_manager.go:414] "containerMap: removing container" podUID="27e0a0a8-ddd7-4d22-91cf-e11612132492" containerName="revoke-keys"
Oct 30 20:52:10.839192 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:10.839133    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e0a0a8-ddd7-4d22-91cf-e11612132492" containerName="revoke-keys"
Oct 30 20:52:10.839271 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:10.839262    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e0a0a8-ddd7-4d22-91cf-e11612132492" containerName="revoke-keys"
Oct 30 20:52:10.843009 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:10.842989    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:10.848752 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:10.848726    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:52:10.848917 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:10.848812    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:52:10.854626 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:10.854596    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"]
Oct 30 20:52:11.003795 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.003755    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f27c36f-kwbsp\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:11.004223 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.004193    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f27c36f-kwbsp\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:11.004384 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.004368    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpv8\" (UniqueName: \"kubernetes.io/projected/ea7f6914-8bc7-44a5-b273-2374b8829692-kube-api-access-hhpv8\") pod \"maas-api-revoke-sub-1f27c36f-kwbsp\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:11.105409 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.105323    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f27c36f-kwbsp\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:11.105409 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.105388    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpv8\" (UniqueName: \"kubernetes.io/projected/ea7f6914-8bc7-44a5-b273-2374b8829692-kube-api-access-hhpv8\") pod \"maas-api-revoke-sub-1f27c36f-kwbsp\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:11.105641 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.105548    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f27c36f-kwbsp\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:11.106060 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.106026    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f27c36f-kwbsp\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:11.108396 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.108370    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f27c36f-kwbsp\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:11.115802 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.115778    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpv8\" (UniqueName: \"kubernetes.io/projected/ea7f6914-8bc7-44a5-b273-2374b8829692-kube-api-access-hhpv8\") pod \"maas-api-revoke-sub-1f27c36f-kwbsp\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:11.165697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.165669    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:11.478913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.465960    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"]
Oct 30 20:52:11.505463 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:52:11.505412    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7f6914_8bc7_44a5_b273_2374b8829692.slice/crio-b81f0320f543abc80f568b091aa116366ab21e904689eff9b4c43a319da1c670 WatchSource:0}: Error finding container b81f0320f543abc80f568b091aa116366ab21e904689eff9b4c43a319da1c670: Status 404 returned error can't find the container with id b81f0320f543abc80f568b091aa116366ab21e904689eff9b4c43a319da1c670
Oct 30 20:52:11.669111 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:11.669071    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp" event={"ID":"ea7f6914-8bc7-44a5-b273-2374b8829692","Type":"ContainerStarted","Data":"b81f0320f543abc80f568b091aa116366ab21e904689eff9b4c43a319da1c670"}
Oct 30 20:52:12.706616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:12.706552    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp" event={"ID":"ea7f6914-8bc7-44a5-b273-2374b8829692","Type":"ContainerStarted","Data":"68e86cda5c60b087e61506cd83a958d1849277c1945c8d0db9d08ea18092e9df"}
Oct 30 20:52:12.756883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:12.755421    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp" podStartSLOduration=2.755389971 podStartE2EDuration="2.755389971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:12.725241202 +0000 UTC m=+899.217057747" watchObservedRunningTime="2026-10-30 20:52:12.755389971 +0000 UTC m=+899.247206518"
Oct 30 20:52:13.108369 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.105555    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"]
Oct 30 20:52:13.124977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.123155    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.128365 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.128293    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"]
Oct 30 20:52:13.204943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.136192    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8e4acb-openshift-default-dockercfg-w4qvm\""
Oct 30 20:52:13.204943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.200067    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfgs\" (UniqueName: \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-kube-api-access-mtfgs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.204943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.200133    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.204943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.200180    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.204943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.200271    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.204943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.200320    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.204943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.200358    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.204943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.200479    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.204943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.200529    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.204943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.200571    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.311503 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.311467    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.311797 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.311774    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.311963 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.311946    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.312184 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.312167    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.312322 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.312306    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.312452 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.312437    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.312566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.312550    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfgs\" (UniqueName: \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-kube-api-access-mtfgs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.312696 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.312679    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.312819 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.312803    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.336880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.313370    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.345884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.337918    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.345884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.339057    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.345884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.339314    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.345884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.339997    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.410886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.408762    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.435246 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.435215    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.436295 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.435900    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.450573 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.450542    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfgs\" (UniqueName: \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-kube-api-access-mtfgs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.467095 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.467068    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:13.975331 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:13.975281    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:14.064656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:14.064610    4810 generic.go:356] "Generic (PLEG): container finished" podID="ea7f6914-8bc7-44a5-b273-2374b8829692" containerID="68e86cda5c60b087e61506cd83a958d1849277c1945c8d0db9d08ea18092e9df" exitCode=0
Oct 30 20:52:14.065284 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:14.065257    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp" event={"ID":"ea7f6914-8bc7-44a5-b273-2374b8829692","Type":"ContainerDied","Data":"68e86cda5c60b087e61506cd83a958d1849277c1945c8d0db9d08ea18092e9df"}
Oct 30 20:52:14.174914 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:14.160963    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:52:14.208178 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:14.208152    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:52:14.239929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:14.239901    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"]
Oct 30 20:52:14.241616 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:52:14.241584    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d6c287_ec16_4df8_a8d0_c57aa9bbaa2e.slice/crio-f163859a38365906fc868573a9b63d8370cb38dd94864f579ab1baa6304b1030 WatchSource:0}: Error finding container f163859a38365906fc868573a9b63d8370cb38dd94864f579ab1baa6304b1030: Status 404 returned error can't find the container with id f163859a38365906fc868573a9b63d8370cb38dd94864f579ab1baa6304b1030
Oct 30 20:52:14.251941 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:14.244131    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:52:14.251941 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:14.244240    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:52:14.251941 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:14.244286    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:52:15.074792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.074736    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" event={"ID":"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e","Type":"ContainerStarted","Data":"3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f"}
Oct 30 20:52:15.075322 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.074812    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" event={"ID":"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e","Type":"ContainerStarted","Data":"f163859a38365906fc868573a9b63d8370cb38dd94864f579ab1baa6304b1030"}
Oct 30 20:52:15.111566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.110653    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" podStartSLOduration=2.110601022 podStartE2EDuration="2.110601022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:15.106666206 +0000 UTC m=+901.598482752" watchObservedRunningTime="2026-10-30 20:52:15.110601022 +0000 UTC m=+901.602417569"
Oct 30 20:52:15.331776 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.331712    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:15.348106 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.348077    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-xks-ca\") pod \"ea7f6914-8bc7-44a5-b273-2374b8829692\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") "
Oct 30 20:52:15.348239 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.348170    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7f6914-8bc7-44a5-b273-2374b8829692-kube-api-access-hhpv8\" (UniqueName: \"kubernetes.io/projected/ea7f6914-8bc7-44a5-b273-2374b8829692-kube-api-access-hhpv8\") pod \"ea7f6914-8bc7-44a5-b273-2374b8829692\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") "
Oct 30 20:52:15.348944 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.348431    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-openshift-ca\") pod \"ea7f6914-8bc7-44a5-b273-2374b8829692\" (UID: \"ea7f6914-8bc7-44a5-b273-2374b8829692\") "
Oct 30 20:52:15.349337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.349312    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-openshift-ca" pod "ea7f6914-8bc7-44a5-b273-2374b8829692" (UID: "ea7f6914-8bc7-44a5-b273-2374b8829692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:15.351065 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.350986    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-xks-ca" pod "ea7f6914-8bc7-44a5-b273-2374b8829692" (UID: "ea7f6914-8bc7-44a5-b273-2374b8829692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:15.352247 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.352220    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7f6914-8bc7-44a5-b273-2374b8829692-kube-api-access-hhpv8" pod "ea7f6914-8bc7-44a5-b273-2374b8829692" (UID: "ea7f6914-8bc7-44a5-b273-2374b8829692"). InnerVolumeSpecName "kube-api-access-hhpv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:15.449929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.449896    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:15.449929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.449920    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhpv8\" (UniqueName: \"kubernetes.io/projected/ea7f6914-8bc7-44a5-b273-2374b8829692-kube-api-access-hhpv8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:15.449929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.449932    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea7f6914-8bc7-44a5-b273-2374b8829692-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:15.467619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.467579    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:15.468972 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.468941    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:52:15.469114 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.468999    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:52:15.957971 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.957898    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"]
Oct 30 20:52:15.958473 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.958459    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ea7f6914-8bc7-44a5-b273-2374b8829692" containerName="revoke-keys"
Oct 30 20:52:15.958512 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.958475    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7f6914-8bc7-44a5-b273-2374b8829692" containerName="revoke-keys"
Oct 30 20:52:15.958597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.958588    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7f6914-8bc7-44a5-b273-2374b8829692" containerName="revoke-keys"
Oct 30 20:52:15.962070 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.962053    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:15.984178 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:15.984144    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"]
Oct 30 20:52:16.058276 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.058239    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxlp6\" (UniqueName: \"kubernetes.io/projected/8c04e81c-8d87-4e08-abc6-ce0146d66535-kube-api-access-bxlp6\") pod \"maas-api-revoke-sub-541db547-8kpgv\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:16.058457 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.058310    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541db547-8kpgv\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:16.058457 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.058370    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541db547-8kpgv\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:16.117816 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.117061    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp" event={"ID":"ea7f6914-8bc7-44a5-b273-2374b8829692","Type":"ContainerDied","Data":"b81f0320f543abc80f568b091aa116366ab21e904689eff9b4c43a319da1c670"}
Oct 30 20:52:16.117816 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.117146    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"
Oct 30 20:52:16.117816 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.117146    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b81f0320f543abc80f568b091aa116366ab21e904689eff9b4c43a319da1c670"
Oct 30 20:52:16.160006 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.159972    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541db547-8kpgv\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:16.160006 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.160017    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541db547-8kpgv\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:16.160302 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.160279    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxlp6\" (UniqueName: \"kubernetes.io/projected/8c04e81c-8d87-4e08-abc6-ce0146d66535-kube-api-access-bxlp6\") pod \"maas-api-revoke-sub-541db547-8kpgv\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:16.160619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.160584    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541db547-8kpgv\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:16.162615 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.162596    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541db547-8kpgv\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:16.170238 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.170218    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxlp6\" (UniqueName: \"kubernetes.io/projected/8c04e81c-8d87-4e08-abc6-ce0146d66535-kube-api-access-bxlp6\") pod \"maas-api-revoke-sub-541db547-8kpgv\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:16.302153 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.302067    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:16.468002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.467964    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:52:16.468179 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.468013    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:52:16.630851 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:52:16.630817    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c04e81c_8d87_4e08_abc6_ce0146d66535.slice/crio-66005162303e37178c6a3ae9af1571997c5248da5912ee7bb31bbe545dbfa729 WatchSource:0}: Error finding container 66005162303e37178c6a3ae9af1571997c5248da5912ee7bb31bbe545dbfa729: Status 404 returned error can't find the container with id 66005162303e37178c6a3ae9af1571997c5248da5912ee7bb31bbe545dbfa729
Oct 30 20:52:16.631011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:16.630988    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"]
Oct 30 20:52:17.157342 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.157308    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv" event={"ID":"8c04e81c-8d87-4e08-abc6-ce0146d66535","Type":"ContainerStarted","Data":"55f085ef5a6c220e41a2041d2446a6792e573615ab7c3c7929ae4acc39ed0c10"}
Oct 30 20:52:17.157705 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.157364    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv" event={"ID":"8c04e81c-8d87-4e08-abc6-ce0146d66535","Type":"ContainerStarted","Data":"66005162303e37178c6a3ae9af1571997c5248da5912ee7bb31bbe545dbfa729"}
Oct 30 20:52:17.308887 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.308057    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv" podStartSLOduration=2.308038117 podStartE2EDuration="2.308038117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:17.185015267 +0000 UTC m=+903.676831814" watchObservedRunningTime="2026-10-30 20:52:17.308038117 +0000 UTC m=+903.799854663"
Oct 30 20:52:17.309650 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.309447    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"]
Oct 30 20:52:17.316289 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.315751    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.318628 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.318594    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"]
Oct 30 20:52:17.402973 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.397589    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69bc17ee-4r46z\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.402973 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.397752    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69bc17ee-4r46z\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.402973 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.397802    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl7mf\" (UniqueName: \"kubernetes.io/projected/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-kube-api-access-cl7mf\") pod \"maas-api-revoke-sub-69bc17ee-4r46z\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.467884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.467826    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:52:17.468022 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.467909    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:52:17.500886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.500623    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69bc17ee-4r46z\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.500886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.500695    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl7mf\" (UniqueName: \"kubernetes.io/projected/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-kube-api-access-cl7mf\") pod \"maas-api-revoke-sub-69bc17ee-4r46z\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.500886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.500825    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69bc17ee-4r46z\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.510918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.501734    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69bc17ee-4r46z\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.511884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.511836    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69bc17ee-4r46z\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.515487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.515466    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl7mf\" (UniqueName: \"kubernetes.io/projected/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-kube-api-access-cl7mf\") pod \"maas-api-revoke-sub-69bc17ee-4r46z\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.633436 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.633405    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:17.954043 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:52:17.954014    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec789b8_d0e5_45cd_9095_bd988b1d78f4.slice/crio-b95371bd0e88898f72160db0fe3030d9906e5fb7d89b1b05a8632c008fde061e WatchSource:0}: Error finding container b95371bd0e88898f72160db0fe3030d9906e5fb7d89b1b05a8632c008fde061e: Status 404 returned error can't find the container with id b95371bd0e88898f72160db0fe3030d9906e5fb7d89b1b05a8632c008fde061e
Oct 30 20:52:17.954164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:17.954143    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"]
Oct 30 20:52:18.165333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:18.165298    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z" event={"ID":"7ec789b8-d0e5-45cd-9095-bd988b1d78f4","Type":"ContainerStarted","Data":"b95371bd0e88898f72160db0fe3030d9906e5fb7d89b1b05a8632c008fde061e"}
Oct 30 20:52:18.166712 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:18.166688    4810 generic.go:356] "Generic (PLEG): container finished" podID="8c04e81c-8d87-4e08-abc6-ce0146d66535" containerID="55f085ef5a6c220e41a2041d2446a6792e573615ab7c3c7929ae4acc39ed0c10" exitCode=0
Oct 30 20:52:18.166830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:18.166770    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv" event={"ID":"8c04e81c-8d87-4e08-abc6-ce0146d66535","Type":"ContainerDied","Data":"55f085ef5a6c220e41a2041d2446a6792e573615ab7c3c7929ae4acc39ed0c10"}
Oct 30 20:52:18.467942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:18.467906    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:52:18.468098 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:18.467960    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:52:19.179766 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.179730    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z" event={"ID":"7ec789b8-d0e5-45cd-9095-bd988b1d78f4","Type":"ContainerStarted","Data":"882f543620d24ee0fe1bc7e964856a0c32c74c9c15de0099237c484867ca3afa"}
Oct 30 20:52:19.215260 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.213601    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z" podStartSLOduration=2.213579833 podStartE2EDuration="2.213579833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:19.198846248 +0000 UTC m=+905.690662794" watchObservedRunningTime="2026-10-30 20:52:19.213579833 +0000 UTC m=+905.705396379"
Oct 30 20:52:19.430621 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.430571    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:19.468298 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.468268    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:52:19.468464 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.468327    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:52:19.537501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.537468    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-openshift-ca\") pod \"8c04e81c-8d87-4e08-abc6-ce0146d66535\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") "
Oct 30 20:52:19.537697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.537634    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c04e81c-8d87-4e08-abc6-ce0146d66535-kube-api-access-bxlp6\" (UniqueName: \"kubernetes.io/projected/8c04e81c-8d87-4e08-abc6-ce0146d66535-kube-api-access-bxlp6\") pod \"8c04e81c-8d87-4e08-abc6-ce0146d66535\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") "
Oct 30 20:52:19.537697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.537690    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-xks-ca\") pod \"8c04e81c-8d87-4e08-abc6-ce0146d66535\" (UID: \"8c04e81c-8d87-4e08-abc6-ce0146d66535\") "
Oct 30 20:52:19.537951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.537928    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-openshift-ca" pod "8c04e81c-8d87-4e08-abc6-ce0146d66535" (UID: "8c04e81c-8d87-4e08-abc6-ce0146d66535"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:19.538192 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.538174    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:19.540004 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.539980    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-xks-ca" pod "8c04e81c-8d87-4e08-abc6-ce0146d66535" (UID: "8c04e81c-8d87-4e08-abc6-ce0146d66535"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:19.540128 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.540104    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c04e81c-8d87-4e08-abc6-ce0146d66535-kube-api-access-bxlp6" pod "8c04e81c-8d87-4e08-abc6-ce0146d66535" (UID: "8c04e81c-8d87-4e08-abc6-ce0146d66535"). InnerVolumeSpecName "kube-api-access-bxlp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:19.639579 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.639547    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxlp6\" (UniqueName: \"kubernetes.io/projected/8c04e81c-8d87-4e08-abc6-ce0146d66535-kube-api-access-bxlp6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:19.639579 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:19.639575    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c04e81c-8d87-4e08-abc6-ce0146d66535-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:20.213094 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:20.213061    4810 generic.go:356] "Generic (PLEG): container finished" podID="7ec789b8-d0e5-45cd-9095-bd988b1d78f4" containerID="882f543620d24ee0fe1bc7e964856a0c32c74c9c15de0099237c484867ca3afa" exitCode=0
Oct 30 20:52:20.213583 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:20.213142    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z" event={"ID":"7ec789b8-d0e5-45cd-9095-bd988b1d78f4","Type":"ContainerDied","Data":"882f543620d24ee0fe1bc7e964856a0c32c74c9c15de0099237c484867ca3afa"}
Oct 30 20:52:20.214695 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:20.214673    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"
Oct 30 20:52:20.214837 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:20.214677    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv" event={"ID":"8c04e81c-8d87-4e08-abc6-ce0146d66535","Type":"ContainerDied","Data":"66005162303e37178c6a3ae9af1571997c5248da5912ee7bb31bbe545dbfa729"}
Oct 30 20:52:20.214837 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:20.214801    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66005162303e37178c6a3ae9af1571997c5248da5912ee7bb31bbe545dbfa729"
Oct 30 20:52:20.468524 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:20.468435    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:52:20.468524 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:20.468494    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:52:21.498926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.497574    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:52:21.498926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.497631    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 20:52:21.543879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.526033    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:21.586521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.584617    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-kube-api-access-cl7mf\" (UniqueName: \"kubernetes.io/projected/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-kube-api-access-cl7mf\") pod \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") "
Oct 30 20:52:21.586521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.584739    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-xks-ca\") pod \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") "
Oct 30 20:52:21.586521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.584828    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-openshift-ca\") pod \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\" (UID: \"7ec789b8-d0e5-45cd-9095-bd988b1d78f4\") "
Oct 30 20:52:21.591128 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.590661    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-openshift-ca" pod "7ec789b8-d0e5-45cd-9095-bd988b1d78f4" (UID: "7ec789b8-d0e5-45cd-9095-bd988b1d78f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:21.613239 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.611365    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-kube-api-access-cl7mf" pod "7ec789b8-d0e5-45cd-9095-bd988b1d78f4" (UID: "7ec789b8-d0e5-45cd-9095-bd988b1d78f4"). InnerVolumeSpecName "kube-api-access-cl7mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:21.613533 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.613450    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-xks-ca" pod "7ec789b8-d0e5-45cd-9095-bd988b1d78f4" (UID: "7ec789b8-d0e5-45cd-9095-bd988b1d78f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:21.687538 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.687496    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl7mf\" (UniqueName: \"kubernetes.io/projected/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-kube-api-access-cl7mf\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:21.687538 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.687537    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:21.687778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:21.687551    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec789b8-d0e5-45cd-9095-bd988b1d78f4-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:22.255891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:22.254010    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"
Oct 30 20:52:22.255891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:22.254051    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z" event={"ID":"7ec789b8-d0e5-45cd-9095-bd988b1d78f4","Type":"ContainerDied","Data":"b95371bd0e88898f72160db0fe3030d9906e5fb7d89b1b05a8632c008fde061e"}
Oct 30 20:52:22.255891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:22.254107    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b95371bd0e88898f72160db0fe3030d9906e5fb7d89b1b05a8632c008fde061e"
Oct 30 20:52:22.473393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:22.473354    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:22.474128 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:22.474104    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:22.474994 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:22.474974    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 20:52:25.162121 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:52:25.162076    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:52:28.150189 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.150152    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"]
Oct 30 20:52:28.150953 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.150931    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8c04e81c-8d87-4e08-abc6-ce0146d66535" containerName="revoke-keys"
Oct 30 20:52:28.150953 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.150950    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c04e81c-8d87-4e08-abc6-ce0146d66535" containerName="revoke-keys"
Oct 30 20:52:28.151135 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.150986    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7ec789b8-d0e5-45cd-9095-bd988b1d78f4" containerName="revoke-keys"
Oct 30 20:52:28.151135 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.150993    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec789b8-d0e5-45cd-9095-bd988b1d78f4" containerName="revoke-keys"
Oct 30 20:52:28.151228 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.151178    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec789b8-d0e5-45cd-9095-bd988b1d78f4" containerName="revoke-keys"
Oct 30 20:52:28.151228 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.151203    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c04e81c-8d87-4e08-abc6-ce0146d66535" containerName="revoke-keys"
Oct 30 20:52:28.156771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.156747    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.176975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.176949    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"]
Oct 30 20:52:28.279381 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.279351    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88286263-1069-4357-a702-95d0a60e6db7-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.279551 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.279420    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.279610 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.279549    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.279610 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.279600    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.279733 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.279683    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88286263-1069-4357-a702-95d0a60e6db7-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.279784 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.279746    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.279918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.279900    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.279980 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.279932    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbnjn\" (UniqueName: \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-kube-api-access-zbnjn\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.280049 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.280036    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.381581 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.381544    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.381751 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.381615    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.381751 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.381647    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.381751 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.381684    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88286263-1069-4357-a702-95d0a60e6db7-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.381751 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.381710    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.381986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.381794    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.381986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.381823    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnjn\" (UniqueName: \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-kube-api-access-zbnjn\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.381986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.381896    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.382138 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.381998    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88286263-1069-4357-a702-95d0a60e6db7-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.382138 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.382040    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.382234 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.382173    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.382482 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.382456    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.382482 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.382472    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88286263-1069-4357-a702-95d0a60e6db7-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.382606 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.382480    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.384795 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.384766    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.384968 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.384950    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88286263-1069-4357-a702-95d0a60e6db7-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.400129 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.400102    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.400674 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.400626    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnjn\" (UniqueName: \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-kube-api-access-zbnjn\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.472022 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.471991    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:28.765284 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.765246    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"]
Oct 30 20:52:28.772834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.772242    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:52:28.772834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.772322    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:52:28.772834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:28.772366    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:52:29.328988 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.328950    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" event={"ID":"88286263-1069-4357-a702-95d0a60e6db7","Type":"ContainerStarted","Data":"64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f"}
Oct 30 20:52:29.329410 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.329012    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" event={"ID":"88286263-1069-4357-a702-95d0a60e6db7","Type":"ContainerStarted","Data":"b0a5d5830d0755759a33428ffa5f677b4314b9113a4510bd74fb677a1f242c6d"}
Oct 30 20:52:29.361480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.361420    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podStartSLOduration=1.361402047 podStartE2EDuration="1.361402047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:29.356232037 +0000 UTC m=+915.848048581" watchObservedRunningTime="2026-10-30 20:52:29.361402047 +0000 UTC m=+915.853218594"
Oct 30 20:52:29.472577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.472537    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:29.474058 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.474033    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:29.474168 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.474089    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:29.624873 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.624762    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"]
Oct 30 20:52:29.629080 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.629053    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.647907 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.646229    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"]
Oct 30 20:52:29.647907 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.646426    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-53aa47-openshift-default-dockercfg-flpcg\""
Oct 30 20:52:29.719880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.716774    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-certs\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.763804 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.763765    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bb7748-246a-49db-9390-9ea9924af937-istio-podinfo\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.763984 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.763877    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.764069 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.763998    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-credential-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.764129 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.764082    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-data\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.764129 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.764115    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-envoy\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.764232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.764213    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-istio-token\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.764291 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.764277    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz9kk\" (UniqueName: \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-kube-api-access-zz9kk\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.764336 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.764316    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bb7748-246a-49db-9390-9ea9924af937-istiod-ca-cert\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865179    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz9kk\" (UniqueName: \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-kube-api-access-zz9kk\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865213    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bb7748-246a-49db-9390-9ea9924af937-istiod-ca-cert\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865452 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865247    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-certs\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865452 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865284    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bb7748-246a-49db-9390-9ea9924af937-istio-podinfo\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865452 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865353    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865452 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865402    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-credential-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865643 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865462    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-data\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865643 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865495    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-envoy\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865643 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865552    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-istio-token\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865760 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865690    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-certs\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.865902 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.865880    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.866048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.866006    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bb7748-246a-49db-9390-9ea9924af937-istiod-ca-cert\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.866184 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.866109    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-data\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.866184 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.866166    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-credential-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.868175 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.868152    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-envoy\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.891078 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.882478    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bb7748-246a-49db-9390-9ea9924af937-istio-podinfo\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.920880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.917572    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz9kk\" (UniqueName: \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-kube-api-access-zz9kk\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.921085 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.918359    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-istio-token\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:29.963230 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.963194    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"]
Oct 30 20:52:29.968501 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.968481    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.004885 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:29.993413    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:30.012877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.011174    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:52:30.012877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.011243    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:52:30.028172 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.028147    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"]
Oct 30 20:52:30.068904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.068460    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bb216e4-sdsd4\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.068904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.068511    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bb216e4-sdsd4\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.068904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.068748    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwc6g\" (UniqueName: \"kubernetes.io/projected/62adfaf2-b44c-475e-91de-91d430e26a2c-kube-api-access-kwc6g\") pod \"maas-api-revoke-sub-8bb216e4-sdsd4\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.169977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.169883    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwc6g\" (UniqueName: \"kubernetes.io/projected/62adfaf2-b44c-475e-91de-91d430e26a2c-kube-api-access-kwc6g\") pod \"maas-api-revoke-sub-8bb216e4-sdsd4\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.227505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.223221    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwc6g\" (UniqueName: \"kubernetes.io/projected/62adfaf2-b44c-475e-91de-91d430e26a2c-kube-api-access-kwc6g\") pod \"maas-api-revoke-sub-8bb216e4-sdsd4\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.227920 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.227453    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bb216e4-sdsd4\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.228088 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.228071    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bb216e4-sdsd4\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.229442 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.229422    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bb216e4-sdsd4\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.231688 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.231657    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bb216e4-sdsd4\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.305017 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.304686    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:30.532350 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.532183    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:30.533542 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.533479    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:30.539136 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.539096    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"]
Oct 30 20:52:30.553066 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.551839    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:52:30.553066 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.552089    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:52:30.553066 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.552171    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:52:30.650447 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:30.650395    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"]
Oct 30 20:52:31.447204 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.447154    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:52:31.453148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.452883    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" event={"ID":"95bb7748-246a-49db-9390-9ea9924af937","Type":"ContainerStarted","Data":"fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974"}
Oct 30 20:52:31.453148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.452957    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" event={"ID":"95bb7748-246a-49db-9390-9ea9924af937","Type":"ContainerStarted","Data":"03b545900e6d0b0a3d4185e0b5249869cbf823b8c5c80e5f585ff338db9772f9"}
Oct 30 20:52:31.457194 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.457153    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4" event={"ID":"62adfaf2-b44c-475e-91de-91d430e26a2c","Type":"ContainerStarted","Data":"604e3797c6f7a0f05952171990e9046f375223e76019f0d9f6934fe714434dd8"}
Oct 30 20:52:31.457488 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.457466    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4" event={"ID":"62adfaf2-b44c-475e-91de-91d430e26a2c","Type":"ContainerStarted","Data":"d22a35bf86469991b4d0c75fef09f70a654ce5ac4afa7d5f6aadc0bdd09d9121"}
Oct 30 20:52:31.480898 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.475829    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:31.481306 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.481268    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:31.534916 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.534825    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podStartSLOduration=2.534789698 podStartE2EDuration="2.534789698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:31.534053532 +0000 UTC m=+918.025870080" watchObservedRunningTime="2026-10-30 20:52:31.534789698 +0000 UTC m=+918.026606246"
Oct 30 20:52:31.561689 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.561641    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4" podStartSLOduration=2.561628277 podStartE2EDuration="2.561628277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:31.557590263 +0000 UTC m=+918.049406821" watchObservedRunningTime="2026-10-30 20:52:31.561628277 +0000 UTC m=+918.053444870"
Oct 30 20:52:31.996895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.995045    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:31.997112 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.997030    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:31.997112 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:31.997084    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:32.481722 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:32.481011    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:32.481722 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:32.481088    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:32.627882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:32.627070    4810 generic.go:356] "Generic (PLEG): container finished" podID="62adfaf2-b44c-475e-91de-91d430e26a2c" containerID="604e3797c6f7a0f05952171990e9046f375223e76019f0d9f6934fe714434dd8" exitCode=0
Oct 30 20:52:32.628422 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:32.628393    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4" event={"ID":"62adfaf2-b44c-475e-91de-91d430e26a2c","Type":"ContainerDied","Data":"604e3797c6f7a0f05952171990e9046f375223e76019f0d9f6934fe714434dd8"}
Oct 30 20:52:33.027883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.005257    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:33.027883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.005332    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:33.492884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.490968    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:33.492884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.491031    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:33.521904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.521870    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"]
Oct 30 20:52:33.886942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.870734    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"]
Oct 30 20:52:33.911881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.892432    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:33.929906 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.918240    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32a20adf-c55f2\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:33.929906 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.918290    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32a20adf-c55f2\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:33.929906 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.918439    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844dk\" (UniqueName: \"kubernetes.io/projected/a81785d4-5c66-4d32-be6e-d443bc4d911c-kube-api-access-844dk\") pod \"maas-api-revoke-sub-32a20adf-c55f2\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:33.981548 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:33.980547    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:34.006025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.003238    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:34.006025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.003296    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:34.022379 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.020934    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32a20adf-c55f2\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:34.022379 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.020989    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32a20adf-c55f2\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:34.022379 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.021131    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844dk\" (UniqueName: \"kubernetes.io/projected/a81785d4-5c66-4d32-be6e-d443bc4d911c-kube-api-access-844dk\") pod \"maas-api-revoke-sub-32a20adf-c55f2\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:34.057895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.054035    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32a20adf-c55f2\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:34.058814 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.058754    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-844dk\" (UniqueName: \"kubernetes.io/projected/a81785d4-5c66-4d32-be6e-d443bc4d911c-kube-api-access-844dk\") pod \"maas-api-revoke-sub-32a20adf-c55f2\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:34.078555 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.078510    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32a20adf-c55f2\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:34.290939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.288384    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:34.325224 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.325195    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:34.473824 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.473787    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:34.475331 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.475302    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:34.523900 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.522951    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:34.544946 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:34.542709    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:35.031026 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.019689    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:35.031026 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.019744    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:35.031026 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.020159    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:35.031026 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.020537    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:52:35.096261 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.096215    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:52:35.301058 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.300961    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:35.307978 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.302013    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:52:35.515799 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.515766    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"]
Oct 30 20:52:35.547157 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.546982    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:35.547629 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.547604    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:52:35.547878 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.547840    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:35.556452 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.556375    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:35.643637 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.643179    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:35.907279 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.907243    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-openshift-ca\") pod \"62adfaf2-b44c-475e-91de-91d430e26a2c\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") "
Oct 30 20:52:35.908053 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.908028    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62adfaf2-b44c-475e-91de-91d430e26a2c-kube-api-access-kwc6g\" (UniqueName: \"kubernetes.io/projected/62adfaf2-b44c-475e-91de-91d430e26a2c-kube-api-access-kwc6g\") pod \"62adfaf2-b44c-475e-91de-91d430e26a2c\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") "
Oct 30 20:52:35.908302 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.908283    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-xks-ca\") pod \"62adfaf2-b44c-475e-91de-91d430e26a2c\" (UID: \"62adfaf2-b44c-475e-91de-91d430e26a2c\") "
Oct 30 20:52:35.909234 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.907922    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-openshift-ca" pod "62adfaf2-b44c-475e-91de-91d430e26a2c" (UID: "62adfaf2-b44c-475e-91de-91d430e26a2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:35.958882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.956446    4810 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:52:35.958882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.956599    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:35.999359 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.999325    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:35.999608 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:35.999582    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:36.030880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.010347    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.031875 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.026389    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:52:36.058880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.040247    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-xks-ca" pod "62adfaf2-b44c-475e-91de-91d430e26a2c" (UID: "62adfaf2-b44c-475e-91de-91d430e26a2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:36.072974 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.068487    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62adfaf2-b44c-475e-91de-91d430e26a2c-kube-api-access-kwc6g" pod "62adfaf2-b44c-475e-91de-91d430e26a2c" (UID: "62adfaf2-b44c-475e-91de-91d430e26a2c"). InnerVolumeSpecName "kube-api-access-kwc6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:36.177124 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.177089    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwc6g\" (UniqueName: \"kubernetes.io/projected/62adfaf2-b44c-475e-91de-91d430e26a2c-kube-api-access-kwc6g\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.179430 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.179405    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62adfaf2-b44c-475e-91de-91d430e26a2c-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.289201 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.289170    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2" event={"ID":"a81785d4-5c66-4d32-be6e-d443bc4d911c","Type":"ContainerStarted","Data":"55f172d1381295ea6fc83ca75f95618d66af5b87aeb0cc3b089d5a0bdb5e1035"}
Oct 30 20:52:36.291933 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.291911    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"
Oct 30 20:52:36.293940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.293841    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4" event={"ID":"62adfaf2-b44c-475e-91de-91d430e26a2c","Type":"ContainerDied","Data":"d22a35bf86469991b4d0c75fef09f70a654ce5ac4afa7d5f6aadc0bdd09d9121"}
Oct 30 20:52:36.293940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.293904    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d22a35bf86469991b4d0c75fef09f70a654ce5ac4afa7d5f6aadc0bdd09d9121"
Oct 30 20:52:36.347792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.346961    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:36.474191 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.474092    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:36.474786 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.474754    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:36.511362 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.511322    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:36.587471 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:36.587420    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 20:52:37.016270 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.016178    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:37.017997 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.017950    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:37.039894 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.032965    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:37.170991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.170948    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:52:37.377931 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.351990    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:37.471342 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.471304    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2" event={"ID":"a81785d4-5c66-4d32-be6e-d443bc4d911c","Type":"ContainerStarted","Data":"8144dfe27baffa6e4ecc735aae90bf45b1e39b288ccb923c531c404541430765"}
Oct 30 20:52:37.473369 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.473342    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:37.473559 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.473536    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:37.606889 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.606796    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:37.706312 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.665626    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2" podStartSLOduration=4.665601322 podStartE2EDuration="4.665601322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:37.536187388 +0000 UTC m=+924.028003936" watchObservedRunningTime="2026-10-30 20:52:37.665601322 +0000 UTC m=+924.157417866"
Oct 30 20:52:37.781231 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:37.781173    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Oct 30 20:52:38.008944 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.003290    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:38.018014 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.017971    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:38.050877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.049919    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:38.050877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.050336    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:52:38.238881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.205919    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:38.262965 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.240016    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:52:38.352578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.352518    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:38.568026 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.558068    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:38.591882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.583600    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:38.591882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.583657    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:38.609924 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.609887    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:38.675938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.651970    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:52:38.675938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.652040    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:38.690428 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.690399    4810 generic.go:356] "Generic (PLEG): container finished" podID="a81785d4-5c66-4d32-be6e-d443bc4d911c" containerID="8144dfe27baffa6e4ecc735aae90bf45b1e39b288ccb923c531c404541430765" exitCode=0
Oct 30 20:52:38.690693 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.690672    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2" event={"ID":"a81785d4-5c66-4d32-be6e-d443bc4d911c","Type":"ContainerDied","Data":"8144dfe27baffa6e4ecc735aae90bf45b1e39b288ccb923c531c404541430765"}
Oct 30 20:52:38.923263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.922942    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 30 20:52:38.999632 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.999202    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:38.999632 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:38.999263    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:39.070915 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:39.070161    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 20:52:39.253235 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:39.250981    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:39.353099 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:39.353049    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:39.438375 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:39.438320    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 20:52:39.444673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:39.444654    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 20:52:39.454533 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:39.454515    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 20:52:39.464097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:39.464077    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 20:52:39.483459 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:39.483433    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:39.483644 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:39.483622    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:40.006928 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.002230    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:40.006928 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.002289    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:40.231560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.231529    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:40.294986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.283301    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-openshift-ca\") pod \"a81785d4-5c66-4d32-be6e-d443bc4d911c\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") "
Oct 30 20:52:40.294986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.283360    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-xks-ca\") pod \"a81785d4-5c66-4d32-be6e-d443bc4d911c\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") "
Oct 30 20:52:40.294986 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.283621    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81785d4-5c66-4d32-be6e-d443bc4d911c-kube-api-access-844dk\" (UniqueName: \"kubernetes.io/projected/a81785d4-5c66-4d32-be6e-d443bc4d911c-kube-api-access-844dk\") pod \"a81785d4-5c66-4d32-be6e-d443bc4d911c\" (UID: \"a81785d4-5c66-4d32-be6e-d443bc4d911c\") "
Oct 30 20:52:40.296572 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.296238    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-openshift-ca" pod "a81785d4-5c66-4d32-be6e-d443bc4d911c" (UID: "a81785d4-5c66-4d32-be6e-d443bc4d911c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:40.322882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.317047    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-xks-ca" pod "a81785d4-5c66-4d32-be6e-d443bc4d911c" (UID: "a81785d4-5c66-4d32-be6e-d443bc4d911c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:40.354219 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.352312    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81785d4-5c66-4d32-be6e-d443bc4d911c-kube-api-access-844dk" pod "a81785d4-5c66-4d32-be6e-d443bc4d911c" (UID: "a81785d4-5c66-4d32-be6e-d443bc4d911c"). InnerVolumeSpecName "kube-api-access-844dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:40.366039 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.366008    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"]
Oct 30 20:52:40.367056 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.367035    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a81785d4-5c66-4d32-be6e-d443bc4d911c" containerName="revoke-keys"
Oct 30 20:52:40.367190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.367180    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81785d4-5c66-4d32-be6e-d443bc4d911c" containerName="revoke-keys"
Oct 30 20:52:40.367274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.367266    4810 cpu_manager.go:414] "containerMap: removing container" podUID="62adfaf2-b44c-475e-91de-91d430e26a2c" containerName="revoke-keys"
Oct 30 20:52:40.367333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.367322    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62adfaf2-b44c-475e-91de-91d430e26a2c" containerName="revoke-keys"
Oct 30 20:52:40.367482 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.367474    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="62adfaf2-b44c-475e-91de-91d430e26a2c" containerName="revoke-keys"
Oct 30 20:52:40.367543 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.367536    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81785d4-5c66-4d32-be6e-d443bc4d911c" containerName="revoke-keys"
Oct 30 20:52:40.423259 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.423227    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"]
Oct 30 20:52:40.423478 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.423313    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.423643 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.423494    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-844dk\" (UniqueName: \"kubernetes.io/projected/a81785d4-5c66-4d32-be6e-d443bc4d911c-kube-api-access-844dk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:40.423809 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.423795    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:40.423939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.423920    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81785d4-5c66-4d32-be6e-d443bc4d911c-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:40.514072 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.514010    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:52:40.514219 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.514151    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 20:52:40.524672 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.524640    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pks7j\" (UniqueName: \"kubernetes.io/projected/d02525b8-208a-43e1-baff-acec730e14f4-kube-api-access-pks7j\") pod \"maas-api-revoke-sub-66e0149d-5tc4t\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.524907 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.524885    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66e0149d-5tc4t\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.525102 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.525071    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66e0149d-5tc4t\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.626774 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.626689    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pks7j\" (UniqueName: \"kubernetes.io/projected/d02525b8-208a-43e1-baff-acec730e14f4-kube-api-access-pks7j\") pod \"maas-api-revoke-sub-66e0149d-5tc4t\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.627531 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.627504    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66e0149d-5tc4t\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.627760 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.627744    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66e0149d-5tc4t\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.628989 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.628849    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66e0149d-5tc4t\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.630737 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.630713    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66e0149d-5tc4t\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.635800 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.635773    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pks7j\" (UniqueName: \"kubernetes.io/projected/d02525b8-208a-43e1-baff-acec730e14f4-kube-api-access-pks7j\") pod \"maas-api-revoke-sub-66e0149d-5tc4t\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.744117 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.744084    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:40.752433 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.752406    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2" event={"ID":"a81785d4-5c66-4d32-be6e-d443bc4d911c","Type":"ContainerDied","Data":"55f172d1381295ea6fc83ca75f95618d66af5b87aeb0cc3b089d5a0bdb5e1035"}
Oct 30 20:52:40.752566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.752455    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55f172d1381295ea6fc83ca75f95618d66af5b87aeb0cc3b089d5a0bdb5e1035"
Oct 30 20:52:40.752566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.752510    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"
Oct 30 20:52:41.014882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.996395    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:41.014882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:40.996458    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:41.169878 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:41.166452    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"]
Oct 30 20:52:41.635023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:41.634961    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:41.635506 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:41.635352    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:41.636470 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:41.636439    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 20:52:41.763288 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:41.763247    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t" event={"ID":"d02525b8-208a-43e1-baff-acec730e14f4","Type":"ContainerStarted","Data":"f7115b7feaa9e83558813fc4309a16ad8f3a14b32c7cc39a93519845b94198cd"}
Oct 30 20:52:41.994194 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:41.994158    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 20:52:41.994401 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:41.994225    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 20:52:42.777011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:42.776937    4810 generic.go:356] "Generic (PLEG): container finished" podID="d02525b8-208a-43e1-baff-acec730e14f4" containerID="860109773f83a21a54692ab3d9ecf363d24716599708108d179f8d62990dd326" exitCode=0
Oct 30 20:52:42.777473 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:42.777030    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t" event={"ID":"d02525b8-208a-43e1-baff-acec730e14f4","Type":"ContainerDied","Data":"860109773f83a21a54692ab3d9ecf363d24716599708108d179f8d62990dd326"}
Oct 30 20:52:43.088938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:43.088465    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:43.092029 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:43.089721    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:43.092029 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:43.091712    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 20:52:43.938291 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:43.938261    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:44.100256 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.100182    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-xks-ca\") pod \"d02525b8-208a-43e1-baff-acec730e14f4\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") "
Oct 30 20:52:44.100256 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.100240    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02525b8-208a-43e1-baff-acec730e14f4-kube-api-access-pks7j\" (UniqueName: \"kubernetes.io/projected/d02525b8-208a-43e1-baff-acec730e14f4-kube-api-access-pks7j\") pod \"d02525b8-208a-43e1-baff-acec730e14f4\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") "
Oct 30 20:52:44.100489 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.100363    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-openshift-ca\") pod \"d02525b8-208a-43e1-baff-acec730e14f4\" (UID: \"d02525b8-208a-43e1-baff-acec730e14f4\") "
Oct 30 20:52:44.100683 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.100660    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-openshift-ca" pod "d02525b8-208a-43e1-baff-acec730e14f4" (UID: "d02525b8-208a-43e1-baff-acec730e14f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:44.100888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.100844    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:44.102674 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.102648    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-xks-ca" pod "d02525b8-208a-43e1-baff-acec730e14f4" (UID: "d02525b8-208a-43e1-baff-acec730e14f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:44.102775 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.102690    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02525b8-208a-43e1-baff-acec730e14f4-kube-api-access-pks7j" pod "d02525b8-208a-43e1-baff-acec730e14f4" (UID: "d02525b8-208a-43e1-baff-acec730e14f4"). InnerVolumeSpecName "kube-api-access-pks7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:44.202303 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.202264    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d02525b8-208a-43e1-baff-acec730e14f4-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:44.202303 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.202299    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pks7j\" (UniqueName: \"kubernetes.io/projected/d02525b8-208a-43e1-baff-acec730e14f4-kube-api-access-pks7j\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:52:44.813882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.811212    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"
Oct 30 20:52:44.813882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.811933    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t" event={"ID":"d02525b8-208a-43e1-baff-acec730e14f4","Type":"ContainerDied","Data":"f7115b7feaa9e83558813fc4309a16ad8f3a14b32c7cc39a93519845b94198cd"}
Oct 30 20:52:44.813882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:44.811986    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7115b7feaa9e83558813fc4309a16ad8f3a14b32c7cc39a93519845b94198cd"
Oct 30 20:52:55.164097 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:52:55.164000    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:52:57.685446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:57.685038    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:52:57.685446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:57.685122    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:58.515908 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:52:58.513651    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:08.633036 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.632941    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"]
Oct 30 20:53:08.635760 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.635731    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d02525b8-208a-43e1-baff-acec730e14f4" containerName="revoke-keys"
Oct 30 20:53:08.635949 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.635932    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02525b8-208a-43e1-baff-acec730e14f4" containerName="revoke-keys"
Oct 30 20:53:08.636243 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.636217    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02525b8-208a-43e1-baff-acec730e14f4" containerName="revoke-keys"
Oct 30 20:53:08.641044 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.641017    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"]
Oct 30 20:53:08.641151 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.641137    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.734571 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.734467    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1fac16-openshift-default-dockercfg-vvhxw\""
Oct 30 20:53:08.767207 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.767154    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.767621 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.767582    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-certs\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.767828 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.767806    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-envoy\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.768197 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.768170    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-credential-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.779219 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.769223    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-data\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.779534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.779509    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gdw\" (UniqueName: \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-kube-api-access-p2gdw\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.780184 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.780158    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-podinfo\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.780644 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.780626    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-token\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.780843 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.780826    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istiod-ca-cert\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882062 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882026    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882062 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882078    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-certs\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882330 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882099    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-envoy\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882330 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882181    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-credential-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882330 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882235    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-data\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882330 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882271    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gdw\" (UniqueName: \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-kube-api-access-p2gdw\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882330 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882306    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-podinfo\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882592 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882362    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-token\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882592 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882402    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istiod-ca-cert\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882611    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882669    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-certs\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.882927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.882901    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-data\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.883214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.883155    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-credential-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.883325 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.883207    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istiod-ca-cert\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.884783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.884763    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-envoy\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.885566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.885539    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-podinfo\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.892596 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.892573    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gdw\" (UniqueName: \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-kube-api-access-p2gdw\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.892794 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.892777    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-token\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:08.977686 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:08.977646    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:09.328413 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:09.328379    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"]
Oct 30 20:53:09.331520 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:09.331078    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:53:09.331520 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:09.331153    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:53:09.331520 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:09.331186    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:53:09.445534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:09.445505    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" event={"ID":"9e0f6ecb-84d8-43f9-add2-2c48115c0fba","Type":"ContainerStarted","Data":"1949af726138c13cef556e48f2123ab66d09a7824765f2dba00997c5e27dde30"}
Oct 30 20:53:10.482890 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:10.482428    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" event={"ID":"9e0f6ecb-84d8-43f9-add2-2c48115c0fba","Type":"ContainerStarted","Data":"a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12"}
Oct 30 20:53:10.991881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:10.984929    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:10.991881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:10.985365    4810 patch_prober.go:29] interesting pod/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 20:53:10.991881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:10.985437    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" podUID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 20:53:11.992284 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:11.992252    4810 patch_prober.go:29] interesting pod/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 20:53:11.992901 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:11.992849    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" podUID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 20:53:12.981884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:12.979290    4810 patch_prober.go:29] interesting pod/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 20:53:12.981884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:12.979355    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" podUID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 20:53:13.259728 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.259600    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" podStartSLOduration=5.25957912 podStartE2EDuration="5.25957912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:10.527020436 +0000 UTC m=+957.018836982" watchObservedRunningTime="2026-10-30 20:53:13.25957912 +0000 UTC m=+959.751395665"
Oct 30 20:53:13.262275 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.261808    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"]
Oct 30 20:53:13.266695 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.266672    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.324231 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.324110    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"]
Oct 30 20:53:13.417772 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.417729    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.418132 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.418107    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfzm\" (UniqueName: \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-kube-api-access-2mfzm\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.418363 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.418345    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.418504 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.418487    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.418615 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.418601    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.418720 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.418706    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.418872 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.418840    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.419022 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.419007    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.419333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.419310    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.523915 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.523762    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.523915 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.523833    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.526465 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.526434    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.526690 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.526668    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.526762 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.526717    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.526847 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.526793    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.526942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.526885    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.527002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.526987    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.527089 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.527073    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.527211 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.527196    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfzm\" (UniqueName: \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-kube-api-access-2mfzm\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.528248 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.528224    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.529327 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.529305    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.529816 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.529797    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.530013 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.529988    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.532480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.532460    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.533223 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.533204    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.536297 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.536275    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfzm\" (UniqueName: \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-kube-api-access-2mfzm\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.537655 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.537631    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.637481 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.637453    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:13.995887 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.995262    4810 patch_prober.go:29] interesting pod/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 20:53:13.995887 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:13.995338    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" podUID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 20:53:14.010534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:14.010368    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"]
Oct 30 20:53:14.014947 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:53:14.014915    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e436e17_6eb0_4c0e_b7b9_ecbbf22c9b30.slice/crio-c361ac7b5dc1137b873a1bea06c445cae347a8befba25ef223e6c38c6efd45a1 WatchSource:0}: Error finding container c361ac7b5dc1137b873a1bea06c445cae347a8befba25ef223e6c38c6efd45a1: Status 404 returned error can't find the container with id c361ac7b5dc1137b873a1bea06c445cae347a8befba25ef223e6c38c6efd45a1
Oct 30 20:53:14.021676 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:14.021648    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:53:14.021825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:14.021811    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:53:14.021951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:14.021937    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 20:53:14.610166 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:14.606019    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" event={"ID":"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30","Type":"ContainerStarted","Data":"d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264"}
Oct 30 20:53:14.610166 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:14.606120    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" event={"ID":"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30","Type":"ContainerStarted","Data":"c361ac7b5dc1137b873a1bea06c445cae347a8befba25ef223e6c38c6efd45a1"}
Oct 30 20:53:14.656213 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:14.656182    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:14.656477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:14.653106    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" podStartSLOduration=1.653087233 podStartE2EDuration="1.653087233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:14.651928085 +0000 UTC m=+961.143744630" watchObservedRunningTime="2026-10-30 20:53:14.653087233 +0000 UTC m=+961.144903778"
Oct 30 20:53:14.982881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:14.978290    4810 patch_prober.go:29] interesting pod/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 20:53:14.982881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:14.978371    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" podUID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 20:53:15.641888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:15.641500    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:53:15.641888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:15.641558    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:53:16.013040 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:16.010046    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:16.013040 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:16.011558    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:16.013497 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:16.013474    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:53:16.638471 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:16.638434    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:53:16.638656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:16.638496    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:53:17.638328 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:17.638298    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:53:17.638699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:17.638362    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:53:18.638676 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:18.638634    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:53:18.639201 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:18.638691    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:53:19.413486 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.413447    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"]
Oct 30 20:53:19.418512 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.418483    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:53:19.419635 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.419611    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"]
Oct 30 20:53:19.426181 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.426158    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"]
Oct 30 20:53:19.426316 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.426276    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:53:19.438899 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.437663    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"]
Oct 30 20:53:19.438899 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.437814    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-84f901-dockercfg-jkbg8\""
Oct 30 20:53:19.438899 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.437869    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-84f901\""
Oct 30 20:53:19.566568 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.546472    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4k8m\" (UniqueName: \"kubernetes.io/projected/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-kube-api-access-g4k8m\") pod \"payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r\" (UID: \"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:53:19.566568 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.546581    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxr7\" (UniqueName: \"kubernetes.io/projected/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-kube-api-access-lfxr7\") pod \"payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw\" (UID: \"0149b3ca-ab00-4c5a-a8a8-549042da1fd8\") " pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:53:19.566568 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.546805    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r\" (UID: \"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:53:19.566568 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.546834    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw\" (UID: \"0149b3ca-ab00-4c5a-a8a8-549042da1fd8\") " pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:53:19.638742 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.638265    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:53:19.638742 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.638325    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:53:19.675940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.656832    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4k8m\" (UniqueName: \"kubernetes.io/projected/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-kube-api-access-g4k8m\") pod \"payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r\" (UID: \"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:53:19.675940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.656958    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxr7\" (UniqueName: \"kubernetes.io/projected/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-kube-api-access-lfxr7\") pod \"payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw\" (UID: \"0149b3ca-ab00-4c5a-a8a8-549042da1fd8\") " pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:53:19.675940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.657144    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r\" (UID: \"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:53:19.675940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.657176    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw\" (UID: \"0149b3ca-ab00-4c5a-a8a8-549042da1fd8\") " pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:53:19.675940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.658216    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw\" (UID: \"0149b3ca-ab00-4c5a-a8a8-549042da1fd8\") " pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:53:19.697513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.691567    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r\" (UID: \"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:53:19.704717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.704647    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4k8m\" (UniqueName: \"kubernetes.io/projected/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-kube-api-access-g4k8m\") pod \"payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r\" (UID: \"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:53:19.750522 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.750180    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:53:19.753648 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.753619    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxr7\" (UniqueName: \"kubernetes.io/projected/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-kube-api-access-lfxr7\") pod \"payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw\" (UID: \"0149b3ca-ab00-4c5a-a8a8-549042da1fd8\") " pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:53:19.758266 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:19.758237    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:53:20.149136 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:20.149107    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"]
Oct 30 20:53:20.149573 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:53:20.149552    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b15a992_c2e3_4cd0_a9f3_a6db6ed2e1d6.slice/crio-b0fa852b4b0ce46906bc08e95d7dd1b491470ec1ac4a2726793c18d30daadeb5 WatchSource:0}: Error finding container b0fa852b4b0ce46906bc08e95d7dd1b491470ec1ac4a2726793c18d30daadeb5: Status 404 returned error can't find the container with id b0fa852b4b0ce46906bc08e95d7dd1b491470ec1ac4a2726793c18d30daadeb5
Oct 30 20:53:20.151521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:20.151504    4810 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:53:20.161479 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:53:20.161455    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0149b3ca_ab00_4c5a_a8a8_549042da1fd8.slice/crio-34a33f7d911e2b7b2a79009074a8d570edbbaf1fe2bb257f8b15d57c334a42c8 WatchSource:0}: Error finding container 34a33f7d911e2b7b2a79009074a8d570edbbaf1fe2bb257f8b15d57c334a42c8: Status 404 returned error can't find the container with id 34a33f7d911e2b7b2a79009074a8d570edbbaf1fe2bb257f8b15d57c334a42c8
Oct 30 20:53:20.161906 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:20.161884    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"]
Oct 30 20:53:20.509453 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:20.509422    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:53:20.663940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:20.655985    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:53:20.663940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:20.656054    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:53:20.760386 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:20.760309    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw" event={"ID":"0149b3ca-ab00-4c5a-a8a8-549042da1fd8","Type":"ContainerStarted","Data":"34a33f7d911e2b7b2a79009074a8d570edbbaf1fe2bb257f8b15d57c334a42c8"}
Oct 30 20:53:20.761529 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:20.761509    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r" event={"ID":"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6","Type":"ContainerStarted","Data":"b0fa852b4b0ce46906bc08e95d7dd1b491470ec1ac4a2726793c18d30daadeb5"}
Oct 30 20:53:21.639603 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:21.638772    4810 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:53:21.639603 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:21.638836    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 20:53:23.062477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:23.061789    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:23.064170 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:23.064089    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:23.065038 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:23.065018    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 20:53:25.156702 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:53:25.156669    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:53:25.833312 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:25.833271    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r" event={"ID":"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6","Type":"ContainerStarted","Data":"be2fe866a439d0640f92cf2e45f6474d4e22261535b28c93f0ebbd411037aaf1"}
Oct 30 20:53:25.833502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:25.833425    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:53:25.834934 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:25.834909    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw" event={"ID":"0149b3ca-ab00-4c5a-a8a8-549042da1fd8","Type":"ContainerStarted","Data":"083dc7192fb6c0100b68b983527ddc4e8e641b89690fac3db1df720243da9d34"}
Oct 30 20:53:25.835069 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:25.834996    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:53:25.853420 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:25.853373    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r" podStartSLOduration=2.133626187 podStartE2EDuration="6.853361506s" totalImagesPullingTime="4.719735319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:25.849901744 +0000 UTC m=+972.341718288" watchObservedRunningTime="2026-10-30 20:53:25.853361506 +0000 UTC m=+972.345178051"
Oct 30 20:53:25.867792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:25.867751    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw" podStartSLOduration=2.10171991 podStartE2EDuration="6.867740814s" totalImagesPullingTime="4.766020904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:25.864824399 +0000 UTC m=+972.356640944" watchObservedRunningTime="2026-10-30 20:53:25.867740814 +0000 UTC m=+972.359557358"
Oct 30 20:53:30.121168 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.121132    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"]
Oct 30 20:53:30.126471 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.126442    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.132337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.132308    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:30.132486 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.132405    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:53:30.135108 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.135081    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"]
Oct 30 20:53:30.234053 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.233990    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.234209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.234135    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdm9\" (UniqueName: \"kubernetes.io/projected/47257709-d927-412c-8dac-0ef241d7a2cb-kube-api-access-lzdm9\") pod \"maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.234265 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.234212    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.336033 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.336001    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.336196 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.336063    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdm9\" (UniqueName: \"kubernetes.io/projected/47257709-d927-412c-8dac-0ef241d7a2cb-kube-api-access-lzdm9\") pod \"maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.336196 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.336159    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.336774 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.336748    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.338443 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.338420    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.344693 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.344668    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdm9\" (UniqueName: \"kubernetes.io/projected/47257709-d927-412c-8dac-0ef241d7a2cb-kube-api-access-lzdm9\") pod \"maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.448342 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.448310    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:30.749212 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.749178    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"]
Oct 30 20:53:30.751036 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:53:30.751011    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47257709_d927_412c_8dac_0ef241d7a2cb.slice/crio-eeeceea2c79056dc935fad520afc816bc3eac59e2cd7e1bade24aa5e0f5e52fc WatchSource:0}: Error finding container eeeceea2c79056dc935fad520afc816bc3eac59e2cd7e1bade24aa5e0f5e52fc: Status 404 returned error can't find the container with id eeeceea2c79056dc935fad520afc816bc3eac59e2cd7e1bade24aa5e0f5e52fc
Oct 30 20:53:30.897801 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:30.897758    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7" event={"ID":"47257709-d927-412c-8dac-0ef241d7a2cb","Type":"ContainerStarted","Data":"eeeceea2c79056dc935fad520afc816bc3eac59e2cd7e1bade24aa5e0f5e52fc"}
Oct 30 20:53:31.906598 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:31.906562    4810 generic.go:356] "Generic (PLEG): container finished" podID="47257709-d927-412c-8dac-0ef241d7a2cb" containerID="5d9a2d5a82e52d4d7fc54015b74fbece530ea392609ead486476232c050962f5" exitCode=0
Oct 30 20:53:31.907008 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:31.906644    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7" event={"ID":"47257709-d927-412c-8dac-0ef241d7a2cb","Type":"ContainerDied","Data":"5d9a2d5a82e52d4d7fc54015b74fbece530ea392609ead486476232c050962f5"}
Oct 30 20:53:32.751062 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.751024    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"]
Oct 30 20:53:32.756343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.756319    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:53:32.762849 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.762827    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0333a9\""
Oct 30 20:53:32.763244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.763221    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0333a9-dockercfg-ndzxh\""
Oct 30 20:53:32.763972 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.763949    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"]
Oct 30 20:53:32.789653 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.789620    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"]
Oct 30 20:53:32.795065 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.795041    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:53:32.819359 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.819331    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"]
Oct 30 20:53:32.911881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.887739    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5\" (UID: \"b9c0902f-9319-477d-a19b-6a0aecd0a1dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:53:32.948750 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.933054    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p45g\" (UniqueName: \"kubernetes.io/projected/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-kube-api-access-5p45g\") pod \"payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6\" (UID: \"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:53:32.948750 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.933120    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh48k\" (UniqueName: \"kubernetes.io/projected/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-kube-api-access-xh48k\") pod \"payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5\" (UID: \"b9c0902f-9319-477d-a19b-6a0aecd0a1dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:53:32.948750 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:32.933193    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6\" (UID: \"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:53:33.036764 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.036729    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6\" (UID: \"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:53:33.037050 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.037028    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5\" (UID: \"b9c0902f-9319-477d-a19b-6a0aecd0a1dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:53:33.037140 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.037081    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p45g\" (UniqueName: \"kubernetes.io/projected/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-kube-api-access-5p45g\") pod \"payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6\" (UID: \"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:53:33.037200 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.037149    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh48k\" (UniqueName: \"kubernetes.io/projected/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-kube-api-access-xh48k\") pod \"payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5\" (UID: \"b9c0902f-9319-477d-a19b-6a0aecd0a1dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:53:33.038402 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.038370    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6\" (UID: \"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:53:33.038507 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.038440    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5\" (UID: \"b9c0902f-9319-477d-a19b-6a0aecd0a1dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:53:33.049586 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.049529    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh48k\" (UniqueName: \"kubernetes.io/projected/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-kube-api-access-xh48k\") pod \"payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5\" (UID: \"b9c0902f-9319-477d-a19b-6a0aecd0a1dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:53:33.059689 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.057885    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p45g\" (UniqueName: \"kubernetes.io/projected/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-kube-api-access-5p45g\") pod \"payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6\" (UID: \"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:53:33.077659 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.077630    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:53:33.113576 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.113538    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:53:33.324625 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.324581    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:33.355695 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.349619    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47257709-d927-412c-8dac-0ef241d7a2cb-kube-api-access-lzdm9\" (UniqueName: \"kubernetes.io/projected/47257709-d927-412c-8dac-0ef241d7a2cb-kube-api-access-lzdm9\") pod \"47257709-d927-412c-8dac-0ef241d7a2cb\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") "
Oct 30 20:53:33.358615 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.358204    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-openshift-ca\") pod \"47257709-d927-412c-8dac-0ef241d7a2cb\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") "
Oct 30 20:53:33.358904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.358874    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-xks-ca\") pod \"47257709-d927-412c-8dac-0ef241d7a2cb\" (UID: \"47257709-d927-412c-8dac-0ef241d7a2cb\") "
Oct 30 20:53:33.359427 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.358585    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-openshift-ca" pod "47257709-d927-412c-8dac-0ef241d7a2cb" (UID: "47257709-d927-412c-8dac-0ef241d7a2cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:33.373120 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.373066    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-xks-ca" pod "47257709-d927-412c-8dac-0ef241d7a2cb" (UID: "47257709-d927-412c-8dac-0ef241d7a2cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:33.373249 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.373128    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47257709-d927-412c-8dac-0ef241d7a2cb-kube-api-access-lzdm9" pod "47257709-d927-412c-8dac-0ef241d7a2cb" (UID: "47257709-d927-412c-8dac-0ef241d7a2cb"). InnerVolumeSpecName "kube-api-access-lzdm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:33.435673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.435636    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"]
Oct 30 20:53:33.462389 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.462359    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzdm9\" (UniqueName: \"kubernetes.io/projected/47257709-d927-412c-8dac-0ef241d7a2cb-kube-api-access-lzdm9\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:53:33.462543 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.462528    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:53:33.462643 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.462631    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47257709-d927-412c-8dac-0ef241d7a2cb-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:53:33.470248 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:33.469529    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"]
Oct 30 20:53:34.068394 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.068314    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"
Oct 30 20:53:34.069127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.068352    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7" event={"ID":"47257709-d927-412c-8dac-0ef241d7a2cb","Type":"ContainerDied","Data":"eeeceea2c79056dc935fad520afc816bc3eac59e2cd7e1bade24aa5e0f5e52fc"}
Oct 30 20:53:34.069127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.069106    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeeceea2c79056dc935fad520afc816bc3eac59e2cd7e1bade24aa5e0f5e52fc"
Oct 30 20:53:34.072513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.072249    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6" event={"ID":"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a","Type":"ContainerStarted","Data":"803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b"}
Oct 30 20:53:34.072513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.072303    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6" event={"ID":"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a","Type":"ContainerStarted","Data":"2a15f58c82186b3cafe1c35baacb7ac64eb07ab4fb39ee930fbdfe3840c2e1f9"}
Oct 30 20:53:34.072513 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.072352    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:53:34.074613 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.074590    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5" event={"ID":"b9c0902f-9319-477d-a19b-6a0aecd0a1dd","Type":"ContainerStarted","Data":"76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf"}
Oct 30 20:53:34.074736 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.074635    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5" event={"ID":"b9c0902f-9319-477d-a19b-6a0aecd0a1dd","Type":"ContainerStarted","Data":"92063ac42c374d2c10f2dfe57abc8482f6dfe6bbad46701a01c97993e9e3813f"}
Oct 30 20:53:34.075115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.074991    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:53:34.111975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.111907    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6" podStartSLOduration=2.11189254 podStartE2EDuration="2.11189254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:34.095181301 +0000 UTC m=+980.586997848" watchObservedRunningTime="2026-10-30 20:53:34.11189254 +0000 UTC m=+980.603709086"
Oct 30 20:53:34.118265 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:34.116446    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5" podStartSLOduration=2.116430082 podStartE2EDuration="2.116430082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:34.110917409 +0000 UTC m=+980.602733955" watchObservedRunningTime="2026-10-30 20:53:34.116430082 +0000 UTC m=+980.608246629"
Oct 30 20:53:36.845579 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:36.845552    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:53:36.846061 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:36.845618    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:53:41.106377 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:41.106338    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:53:45.088138 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:45.088107    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:53:46.097588 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:46.097560    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:53:49.364649 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.364615    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"]
Oct 30 20:53:49.365272 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.365255    4810 cpu_manager.go:414] "containerMap: removing container" podUID="47257709-d927-412c-8dac-0ef241d7a2cb" containerName="revoke-keys"
Oct 30 20:53:49.365272 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.365272    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47257709-d927-412c-8dac-0ef241d7a2cb" containerName="revoke-keys"
Oct 30 20:53:49.365431 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.365421    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="47257709-d927-412c-8dac-0ef241d7a2cb" containerName="revoke-keys"
Oct 30 20:53:49.370591 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.370570    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.376723 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.376700    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:49.377023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.377005    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:53:49.377751 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.377724    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"]
Oct 30 20:53:49.446161 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.446124    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7mq2\" (UniqueName: \"kubernetes.io/projected/91098eef-5a42-4d57-a9a6-f805bc2c6143-kube-api-access-g7mq2\") pod \"maas-api-revoke-keys-e2e-api-a-95a684-zbbtl\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.446309 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.446173    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-95a684-zbbtl\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.446371 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.446349    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-95a684-zbbtl\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.547556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.547521    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-95a684-zbbtl\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.547717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.547679    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-95a684-zbbtl\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.547913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.547850    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7mq2\" (UniqueName: \"kubernetes.io/projected/91098eef-5a42-4d57-a9a6-f805bc2c6143-kube-api-access-g7mq2\") pod \"maas-api-revoke-keys-e2e-api-a-95a684-zbbtl\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.548305 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.548282    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-95a684-zbbtl\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.550113 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.550090    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-95a684-zbbtl\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.556994 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.556975    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7mq2\" (UniqueName: \"kubernetes.io/projected/91098eef-5a42-4d57-a9a6-f805bc2c6143-kube-api-access-g7mq2\") pod \"maas-api-revoke-keys-e2e-api-a-95a684-zbbtl\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.703549 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.703516    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:49.982078 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:49.982047    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"]
Oct 30 20:53:49.982973 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:53:49.982946    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91098eef_5a42_4d57_a9a6_f805bc2c6143.slice/crio-ecb9005bfe8169d05d7e58e3219cd0f206c5ccc7999f62c598bc77db67b94116 WatchSource:0}: Error finding container ecb9005bfe8169d05d7e58e3219cd0f206c5ccc7999f62c598bc77db67b94116: Status 404 returned error can't find the container with id ecb9005bfe8169d05d7e58e3219cd0f206c5ccc7999f62c598bc77db67b94116
Oct 30 20:53:50.283204 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:50.283143    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl" event={"ID":"91098eef-5a42-4d57-a9a6-f805bc2c6143","Type":"ContainerStarted","Data":"ecb9005bfe8169d05d7e58e3219cd0f206c5ccc7999f62c598bc77db67b94116"}
Oct 30 20:53:51.294974 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:51.294882    4810 generic.go:356] "Generic (PLEG): container finished" podID="91098eef-5a42-4d57-a9a6-f805bc2c6143" containerID="95037646660c01003b288eff4d5972d583cc59429a2d259c9497cd6a32d1d3a4" exitCode=0
Oct 30 20:53:51.295378 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:51.295056    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl" event={"ID":"91098eef-5a42-4d57-a9a6-f805bc2c6143","Type":"ContainerDied","Data":"95037646660c01003b288eff4d5972d583cc59429a2d259c9497cd6a32d1d3a4"}
Oct 30 20:53:52.470919 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:52.470891    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:52.592248 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:52.592170    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91098eef-5a42-4d57-a9a6-f805bc2c6143-kube-api-access-g7mq2\" (UniqueName: \"kubernetes.io/projected/91098eef-5a42-4d57-a9a6-f805bc2c6143-kube-api-access-g7mq2\") pod \"91098eef-5a42-4d57-a9a6-f805bc2c6143\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") "
Oct 30 20:53:52.592248 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:52.592210    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-openshift-ca\") pod \"91098eef-5a42-4d57-a9a6-f805bc2c6143\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") "
Oct 30 20:53:52.592248 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:52.592242    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-xks-ca\") pod \"91098eef-5a42-4d57-a9a6-f805bc2c6143\" (UID: \"91098eef-5a42-4d57-a9a6-f805bc2c6143\") "
Oct 30 20:53:52.592634 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:52.592609    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-openshift-ca" pod "91098eef-5a42-4d57-a9a6-f805bc2c6143" (UID: "91098eef-5a42-4d57-a9a6-f805bc2c6143"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:52.592948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:52.592927    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:53:52.594456 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:52.594424    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-xks-ca" pod "91098eef-5a42-4d57-a9a6-f805bc2c6143" (UID: "91098eef-5a42-4d57-a9a6-f805bc2c6143"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:52.594771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:52.594751    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91098eef-5a42-4d57-a9a6-f805bc2c6143-kube-api-access-g7mq2" pod "91098eef-5a42-4d57-a9a6-f805bc2c6143" (UID: "91098eef-5a42-4d57-a9a6-f805bc2c6143"). InnerVolumeSpecName "kube-api-access-g7mq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:52.694046 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:52.694012    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7mq2\" (UniqueName: \"kubernetes.io/projected/91098eef-5a42-4d57-a9a6-f805bc2c6143-kube-api-access-g7mq2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:53:52.694046 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:52.694043    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91098eef-5a42-4d57-a9a6-f805bc2c6143-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.314462 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:53.314429    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"
Oct 30 20:53:53.314637 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:53.314435    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl" event={"ID":"91098eef-5a42-4d57-a9a6-f805bc2c6143","Type":"ContainerDied","Data":"ecb9005bfe8169d05d7e58e3219cd0f206c5ccc7999f62c598bc77db67b94116"}
Oct 30 20:53:53.314637 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:53.314553    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecb9005bfe8169d05d7e58e3219cd0f206c5ccc7999f62c598bc77db67b94116"
Oct 30 20:53:55.162303 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:53:55.162267    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:53:57.589581 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.589547    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"]
Oct 30 20:53:57.590411 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.590387    4810 cpu_manager.go:414] "containerMap: removing container" podUID="91098eef-5a42-4d57-a9a6-f805bc2c6143" containerName="revoke-keys"
Oct 30 20:53:57.590411 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.590411    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91098eef-5a42-4d57-a9a6-f805bc2c6143" containerName="revoke-keys"
Oct 30 20:53:57.590622 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.590609    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="91098eef-5a42-4d57-a9a6-f805bc2c6143" containerName="revoke-keys"
Oct 30 20:53:57.597924 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.597897    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:57.602161 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.602090    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:57.602283 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.602244    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:53:57.603886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.603846    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"]
Oct 30 20:53:57.684171 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.684138    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2qw\" (UniqueName: \"kubernetes.io/projected/7f56755f-347b-47d9-be99-5b72184fdbe8-kube-api-access-tp2qw\") pod \"maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:57.684321 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.684245    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:57.684321 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.684312    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:57.786413 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.786366    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:57.786608 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.786466    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:57.786775 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.786756    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2qw\" (UniqueName: \"kubernetes.io/projected/7f56755f-347b-47d9-be99-5b72184fdbe8-kube-api-access-tp2qw\") pod \"maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:57.790467 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.787342    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:57.790467 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.789678    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:57.798695 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.798666    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2qw\" (UniqueName: \"kubernetes.io/projected/7f56755f-347b-47d9-be99-5b72184fdbe8-kube-api-access-tp2qw\") pod \"maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:57.917146 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:57.917115    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:53:58.210216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:58.210181    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"]
Oct 30 20:53:58.210950 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:53:58.210918    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f56755f_347b_47d9_be99_5b72184fdbe8.slice/crio-c370afc83f9ac0d5ccb82384d432268ccffb0686775914f65be01ca092a2773d WatchSource:0}: Error finding container c370afc83f9ac0d5ccb82384d432268ccffb0686775914f65be01ca092a2773d: Status 404 returned error can't find the container with id c370afc83f9ac0d5ccb82384d432268ccffb0686775914f65be01ca092a2773d
Oct 30 20:53:58.373836 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:58.373324    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr" event={"ID":"7f56755f-347b-47d9-be99-5b72184fdbe8","Type":"ContainerStarted","Data":"c370afc83f9ac0d5ccb82384d432268ccffb0686775914f65be01ca092a2773d"}
Oct 30 20:53:59.385292 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:59.385259    4810 generic.go:356] "Generic (PLEG): container finished" podID="7f56755f-347b-47d9-be99-5b72184fdbe8" containerID="20e1b305c36dbc2c00f5243307cf14d988c928676541e1446b9d79944194316c" exitCode=0
Oct 30 20:53:59.385738 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:53:59.385348    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr" event={"ID":"7f56755f-347b-47d9-be99-5b72184fdbe8","Type":"ContainerDied","Data":"20e1b305c36dbc2c00f5243307cf14d988c928676541e1446b9d79944194316c"}
Oct 30 20:54:00.570208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:00.570184    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:54:00.628370 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:00.628335    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-xks-ca\") pod \"7f56755f-347b-47d9-be99-5b72184fdbe8\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") "
Oct 30 20:54:00.628553 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:00.628528    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f56755f-347b-47d9-be99-5b72184fdbe8-kube-api-access-tp2qw\" (UniqueName: \"kubernetes.io/projected/7f56755f-347b-47d9-be99-5b72184fdbe8-kube-api-access-tp2qw\") pod \"7f56755f-347b-47d9-be99-5b72184fdbe8\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") "
Oct 30 20:54:00.628668 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:00.628650    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-openshift-ca\") pod \"7f56755f-347b-47d9-be99-5b72184fdbe8\" (UID: \"7f56755f-347b-47d9-be99-5b72184fdbe8\") "
Oct 30 20:54:00.629092 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:00.629052    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-openshift-ca" pod "7f56755f-347b-47d9-be99-5b72184fdbe8" (UID: "7f56755f-347b-47d9-be99-5b72184fdbe8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:00.629274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:00.629249    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:00.630546 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:00.630522    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-xks-ca" pod "7f56755f-347b-47d9-be99-5b72184fdbe8" (UID: "7f56755f-347b-47d9-be99-5b72184fdbe8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:00.631109 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:00.631089    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f56755f-347b-47d9-be99-5b72184fdbe8-kube-api-access-tp2qw" pod "7f56755f-347b-47d9-be99-5b72184fdbe8" (UID: "7f56755f-347b-47d9-be99-5b72184fdbe8"). InnerVolumeSpecName "kube-api-access-tp2qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:00.730439 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:00.730359    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f56755f-347b-47d9-be99-5b72184fdbe8-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:00.730611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:00.730589    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp2qw\" (UniqueName: \"kubernetes.io/projected/7f56755f-347b-47d9-be99-5b72184fdbe8-kube-api-access-tp2qw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:01.406205 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:01.406172    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"
Oct 30 20:54:01.406412 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:01.406188    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr" event={"ID":"7f56755f-347b-47d9-be99-5b72184fdbe8","Type":"ContainerDied","Data":"c370afc83f9ac0d5ccb82384d432268ccffb0686775914f65be01ca092a2773d"}
Oct 30 20:54:01.406412 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:01.406287    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c370afc83f9ac0d5ccb82384d432268ccffb0686775914f65be01ca092a2773d"
Oct 30 20:54:02.871411 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:02.871366    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"]
Oct 30 20:54:02.880786 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:02.880750    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1fac16-xnxs7"]
Oct 30 20:54:03.072717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:03.072682    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"]
Oct 30 20:54:03.086326 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:03.086278    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27950da6-tkzft"]
Oct 30 20:54:03.663708 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:03.663676    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47257709-d927-412c-8dac-0ef241d7a2cb" path="/var/lib/kubelet/pods/47257709-d927-412c-8dac-0ef241d7a2cb/volumes"
Oct 30 20:54:03.664195 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:03.664180    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69ce9d3-933d-4fc7-922a-b9755d5d2285" path="/var/lib/kubelet/pods/d69ce9d3-933d-4fc7-922a-b9755d5d2285/volumes"
Oct 30 20:54:07.255577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.255545    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7615-065799"]
Oct 30 20:54:07.256275 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.256259    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7f56755f-347b-47d9-be99-5b72184fdbe8" containerName="revoke-keys"
Oct 30 20:54:07.256319 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.256276    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f56755f-347b-47d9-be99-5b72184fdbe8" containerName="revoke-keys"
Oct 30 20:54:07.256397 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.256388    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f56755f-347b-47d9-be99-5b72184fdbe8" containerName="revoke-keys"
Oct 30 20:54:07.261383 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.261359    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-065799"
Oct 30 20:54:07.264781 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.264752    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7615-065799"]
Oct 30 20:54:07.266047 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.266022    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrzx8\""
Oct 30 20:54:07.321845 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.321815    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plsb\" (UniqueName: \"kubernetes.io/projected/b3baa8c3-93c9-4a97-a988-21f4645e361d-kube-api-access-5plsb\") pod \"test-curl-7615-065799\" (UID: \"b3baa8c3-93c9-4a97-a988-21f4645e361d\") " pod="opendatahub/test-curl-7615-065799"
Oct 30 20:54:07.422974 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.422942    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plsb\" (UniqueName: \"kubernetes.io/projected/b3baa8c3-93c9-4a97-a988-21f4645e361d-kube-api-access-5plsb\") pod \"test-curl-7615-065799\" (UID: \"b3baa8c3-93c9-4a97-a988-21f4645e361d\") " pod="opendatahub/test-curl-7615-065799"
Oct 30 20:54:07.431511 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.431486    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plsb\" (UniqueName: \"kubernetes.io/projected/b3baa8c3-93c9-4a97-a988-21f4645e361d-kube-api-access-5plsb\") pod \"test-curl-7615-065799\" (UID: \"b3baa8c3-93c9-4a97-a988-21f4645e361d\") " pod="opendatahub/test-curl-7615-065799"
Oct 30 20:54:07.577956 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.577832    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-065799"
Oct 30 20:54:07.908307 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:07.907251    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7615-065799"]
Oct 30 20:54:08.483527 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:08.483484    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7615-065799" event={"ID":"b3baa8c3-93c9-4a97-a988-21f4645e361d","Type":"ContainerStarted","Data":"ffd079308d434252dad761cd95c82b6dafb00f1a07433bb3ddf93000ef6d7e11"}
Oct 30 20:54:10.157843 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.157796    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"]
Oct 30 20:54:10.158480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.158123    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" podUID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" containerName="istio-proxy" containerID="cri-o://a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12" gracePeriod=30
Oct 30 20:54:10.681320 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.681285    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"]
Oct 30 20:54:10.687586 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.687562    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:10.699011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.698735    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"]
Oct 30 20:54:10.748771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.746632    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:54:10.748771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.746670    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:54:10.786150 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.786117    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49990a-f7z8n\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:10.786325 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.786191    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59tn\" (UniqueName: \"kubernetes.io/projected/69a2f386-5580-42f4-9066-82a9179735d5-kube-api-access-n59tn\") pod \"maas-api-revoke-keys-e2e-api-b-49990a-f7z8n\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:10.786462 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.786438    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49990a-f7z8n\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:10.887833 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.887800    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49990a-f7z8n\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:10.888012 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.887852    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59tn\" (UniqueName: \"kubernetes.io/projected/69a2f386-5580-42f4-9066-82a9179735d5-kube-api-access-n59tn\") pod \"maas-api-revoke-keys-e2e-api-b-49990a-f7z8n\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:10.888012 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.887977    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49990a-f7z8n\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:10.888839 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.888810    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49990a-f7z8n\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:10.890487 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.890462    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49990a-f7z8n\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:10.897622 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:10.897594    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59tn\" (UniqueName: \"kubernetes.io/projected/69a2f386-5580-42f4-9066-82a9179735d5-kube-api-access-n59tn\") pod \"maas-api-revoke-keys-e2e-api-b-49990a-f7z8n\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:11.018984 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:11.018898    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:11.305211 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:11.305183    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"]
Oct 30 20:54:11.305692 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:54:11.305670    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a2f386_5580_42f4_9066_82a9179735d5.slice/crio-bfceb293739227c078f30ddabd7c08f11a56829c54bde2fd5c68dc8ecd658c45 WatchSource:0}: Error finding container bfceb293739227c078f30ddabd7c08f11a56829c54bde2fd5c68dc8ecd658c45: Status 404 returned error can't find the container with id bfceb293739227c078f30ddabd7c08f11a56829c54bde2fd5c68dc8ecd658c45
Oct 30 20:54:11.513209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:11.513160    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n" event={"ID":"69a2f386-5580-42f4-9066-82a9179735d5","Type":"ContainerStarted","Data":"bfceb293739227c078f30ddabd7c08f11a56829c54bde2fd5c68dc8ecd658c45"}
Oct 30 20:54:12.526512 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:12.526411    4810 generic.go:356] "Generic (PLEG): container finished" podID="69a2f386-5580-42f4-9066-82a9179735d5" containerID="03977cd79ec23daea09ef44cc13845b58b7e838b70cd2038c6d4cd5cbe197501" exitCode=0
Oct 30 20:54:12.526512 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:12.526481    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n" event={"ID":"69a2f386-5580-42f4-9066-82a9179735d5","Type":"ContainerDied","Data":"03977cd79ec23daea09ef44cc13845b58b7e838b70cd2038c6d4cd5cbe197501"}
Oct 30 20:54:13.564961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.564925    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"]
Oct 30 20:54:13.600881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.600256    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.600881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.600455    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"]
Oct 30 20:54:13.738454 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.738418    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmdl\" (UniqueName: \"kubernetes.io/projected/393e6892-57bd-4867-a88a-0ee89441fd04-kube-api-access-9qmdl\") pod \"maas-api-revoke-sub-c3136b91-8kwzv\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.738973 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.738933    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3136b91-8kwzv\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.739210 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.739191    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3136b91-8kwzv\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.815460 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.815391    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:13.840887 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.840837    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmdl\" (UniqueName: \"kubernetes.io/projected/393e6892-57bd-4867-a88a-0ee89441fd04-kube-api-access-9qmdl\") pod \"maas-api-revoke-sub-c3136b91-8kwzv\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.841272 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.841248    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3136b91-8kwzv\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.841407 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.841339    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3136b91-8kwzv\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.841969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.841950    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3136b91-8kwzv\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.844706 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.844662    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3136b91-8kwzv\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.855504 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.855478    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmdl\" (UniqueName: \"kubernetes.io/projected/393e6892-57bd-4867-a88a-0ee89441fd04-kube-api-access-9qmdl\") pod \"maas-api-revoke-sub-c3136b91-8kwzv\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.923619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.923586    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:13.943096 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.943064    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-openshift-ca\") pod \"69a2f386-5580-42f4-9066-82a9179735d5\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") "
Oct 30 20:54:13.943418 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.943396    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-xks-ca\") pod \"69a2f386-5580-42f4-9066-82a9179735d5\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") "
Oct 30 20:54:13.943489 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.943442    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a2f386-5580-42f4-9066-82a9179735d5-kube-api-access-n59tn\" (UniqueName: \"kubernetes.io/projected/69a2f386-5580-42f4-9066-82a9179735d5-kube-api-access-n59tn\") pod \"69a2f386-5580-42f4-9066-82a9179735d5\" (UID: \"69a2f386-5580-42f4-9066-82a9179735d5\") "
Oct 30 20:54:13.944584 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.944559    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-openshift-ca" pod "69a2f386-5580-42f4-9066-82a9179735d5" (UID: "69a2f386-5580-42f4-9066-82a9179735d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:13.946736 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.946699    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a2f386-5580-42f4-9066-82a9179735d5-kube-api-access-n59tn" pod "69a2f386-5580-42f4-9066-82a9179735d5" (UID: "69a2f386-5580-42f4-9066-82a9179735d5"). InnerVolumeSpecName "kube-api-access-n59tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:13.954930 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:13.954907    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-xks-ca" pod "69a2f386-5580-42f4-9066-82a9179735d5" (UID: "69a2f386-5580-42f4-9066-82a9179735d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:14.046667 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.046635    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:14.046667 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.046667    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n59tn\" (UniqueName: \"kubernetes.io/projected/69a2f386-5580-42f4-9066-82a9179735d5-kube-api-access-n59tn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:14.046847 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.046683    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a2f386-5580-42f4-9066-82a9179735d5-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:14.085355 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.085275    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"]
Oct 30 20:54:14.103958 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.103919    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0080b10-2gdsl"]
Oct 30 20:54:14.141137 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.141108    4810 scope.go:122] "RemoveContainer" containerID="786788361e6aa883b38ef8ea44ea5bda490150a5e2e61b567750008b57a620fb"
Oct 30 20:54:14.151640 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.151621    4810 scope.go:122] "RemoveContainer" containerID="712050a65911405e27a2ec8cabb48eb6752bd146d681013ceceda23f57d711ca"
Oct 30 20:54:14.206427 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.206405    4810 scope.go:122] "RemoveContainer" containerID="20035cd676bedfba88509b9cb5f10fca15aada8732e15c60e0c600aa867ef1d3"
Oct 30 20:54:14.218998 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.218937    4810 scope.go:122] "RemoveContainer" containerID="a5a4bb126b158538a56fba12ce33614219f4bf4d78798ad37c1976eb7dec0989"
Oct 30 20:54:14.223187 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:54:14.223163    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393e6892_57bd_4867_a88a_0ee89441fd04.slice/crio-0191986d38805e3143fafe72acf9c7f5bcc9a851080ae915f214221b676f5ebd WatchSource:0}: Error finding container 0191986d38805e3143fafe72acf9c7f5bcc9a851080ae915f214221b676f5ebd: Status 404 returned error can't find the container with id 0191986d38805e3143fafe72acf9c7f5bcc9a851080ae915f214221b676f5ebd
Oct 30 20:54:14.224193 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.224124    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"]
Oct 30 20:54:14.546619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.546584    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7615-065799" event={"ID":"b3baa8c3-93c9-4a97-a988-21f4645e361d","Type":"ContainerStarted","Data":"a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c"}
Oct 30 20:54:14.548045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.548019    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"
Oct 30 20:54:14.548178 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.548064    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n" event={"ID":"69a2f386-5580-42f4-9066-82a9179735d5","Type":"ContainerDied","Data":"bfceb293739227c078f30ddabd7c08f11a56829c54bde2fd5c68dc8ecd658c45"}
Oct 30 20:54:14.548178 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.548104    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfceb293739227c078f30ddabd7c08f11a56829c54bde2fd5c68dc8ecd658c45"
Oct 30 20:54:14.549455 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.549424    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv" event={"ID":"393e6892-57bd-4867-a88a-0ee89441fd04","Type":"ContainerStarted","Data":"0191986d38805e3143fafe72acf9c7f5bcc9a851080ae915f214221b676f5ebd"}
Oct 30 20:54:14.563061 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:14.563013    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7615-065799" podStartSLOduration=1.25988686 podStartE2EDuration="7.562999673s" totalImagesPullingTime="6.303112813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:14.561243145 +0000 UTC m=+1021.053059691" watchObservedRunningTime="2026-10-30 20:54:14.562999673 +0000 UTC m=+1021.054816218"
Oct 30 20:54:15.028219 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.028179    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"]
Oct 30 20:54:15.032344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.032314    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0752c29f-55bx8"]
Oct 30 20:54:15.284312 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.284278    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7615-065799"]
Oct 30 20:54:15.288578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.288550    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7615-065799"]
Oct 30 20:54:15.476949 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.476919    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:54:15.572539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.570112    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-socket\") pod \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") "
Oct 30 20:54:15.572539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.570170    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-certs\") pod \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") "
Oct 30 20:54:15.572539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.570247    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-data\") pod \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") "
Oct 30 20:54:15.572539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.570486    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-kube-api-access-p2gdw\" (UniqueName: \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-kube-api-access-p2gdw\") pod \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") "
Oct 30 20:54:15.572539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.570520    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-token\" (UniqueName: \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-token\") pod \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") "
Oct 30 20:54:15.572539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.570550    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-envoy\") pod \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") "
Oct 30 20:54:15.572539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.570576    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-podinfo\") pod \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") "
Oct 30 20:54:15.572539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.570618    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-credential-socket\") pod \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") "
Oct 30 20:54:15.572539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.570647    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istiod-ca-cert\") pod \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\" (UID: \"9e0f6ecb-84d8-43f9-add2-2c48115c0fba\") "
Oct 30 20:54:15.574798 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.574757    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istiod-ca-cert" pod "9e0f6ecb-84d8-43f9-add2-2c48115c0fba" (UID: "9e0f6ecb-84d8-43f9-add2-2c48115c0fba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:15.575329 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.575307    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-socket" pod "9e0f6ecb-84d8-43f9-add2-2c48115c0fba" (UID: "9e0f6ecb-84d8-43f9-add2-2c48115c0fba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:15.575643 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.575625    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-certs" pod "9e0f6ecb-84d8-43f9-add2-2c48115c0fba" (UID: "9e0f6ecb-84d8-43f9-add2-2c48115c0fba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:15.575960 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.575940    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-data" pod "9e0f6ecb-84d8-43f9-add2-2c48115c0fba" (UID: "9e0f6ecb-84d8-43f9-add2-2c48115c0fba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:15.577353 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.577324    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-credential-socket" pod "9e0f6ecb-84d8-43f9-add2-2c48115c0fba" (UID: "9e0f6ecb-84d8-43f9-add2-2c48115c0fba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:15.579914 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.579678    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-podinfo" pod "9e0f6ecb-84d8-43f9-add2-2c48115c0fba" (UID: "9e0f6ecb-84d8-43f9-add2-2c48115c0fba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:15.581448 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.581402    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-token" pod "9e0f6ecb-84d8-43f9-add2-2c48115c0fba" (UID: "9e0f6ecb-84d8-43f9-add2-2c48115c0fba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:15.585079 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.582038    4810 generic.go:356] "Generic (PLEG): container finished" podID="393e6892-57bd-4867-a88a-0ee89441fd04" containerID="03bf8e05a5b15ad3b1e3297210550b96f4d2726a1063508c94b84f7631594ec2" exitCode=0
Oct 30 20:54:15.585079 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.582254    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv" event={"ID":"393e6892-57bd-4867-a88a-0ee89441fd04","Type":"ContainerDied","Data":"03bf8e05a5b15ad3b1e3297210550b96f4d2726a1063508c94b84f7631594ec2"}
Oct 30 20:54:15.585079 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.584886    4810 generic.go:356] "Generic (PLEG): container finished" podID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" containerID="a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12" exitCode=0
Oct 30 20:54:15.585956 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.585914    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"
Oct 30 20:54:15.586140 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.586117    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" event={"ID":"9e0f6ecb-84d8-43f9-add2-2c48115c0fba","Type":"ContainerDied","Data":"a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12"}
Oct 30 20:54:15.586222 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.586168    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v" event={"ID":"9e0f6ecb-84d8-43f9-add2-2c48115c0fba","Type":"ContainerDied","Data":"1949af726138c13cef556e48f2123ab66d09a7824765f2dba00997c5e27dde30"}
Oct 30 20:54:15.586222 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.586201    4810 scope.go:122] "RemoveContainer" containerID="a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12"
Oct 30 20:54:15.595636 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.595585    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-kube-api-access-p2gdw" pod "9e0f6ecb-84d8-43f9-add2-2c48115c0fba" (UID: "9e0f6ecb-84d8-43f9-add2-2c48115c0fba"). InnerVolumeSpecName "kube-api-access-p2gdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:15.598327 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.598290    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-envoy" pod "9e0f6ecb-84d8-43f9-add2-2c48115c0fba" (UID: "9e0f6ecb-84d8-43f9-add2-2c48115c0fba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:15.611731 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.611702    4810 scope.go:122] "RemoveContainer" containerID="a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12"
Oct 30 20:54:15.612229 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:54:15.612197    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12\": container with ID starting with a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12 not found: ID does not exist" containerID="a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12"
Oct 30 20:54:15.612334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.612231    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12"} err="failed to get container status \"a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12\": rpc error: code = NotFound desc = could not find container \"a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12\": container with ID starting with a6b161d7886a10573d9a48196f394e1ff4d09958c1a255b446b06c50816fee12 not found: ID does not exist"
Oct 30 20:54:15.663886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.663838    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b25634-57cc-4352-af9b-17e10f50978c" path="/var/lib/kubelet/pods/19b25634-57cc-4352-af9b-17e10f50978c/volumes"
Oct 30 20:54:15.664382 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.664367    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9ad61e-d3ae-4607-a169-ee97d925699f" path="/var/lib/kubelet/pods/df9ad61e-d3ae-4607-a169-ee97d925699f/volumes"
Oct 30 20:54:15.676208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.676188    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2gdw\" (UniqueName: \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-kube-api-access-p2gdw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:15.676208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.676209    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:15.676357 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.676242    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:15.676357 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.676253    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:15.676357 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.676263    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:15.676357 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.676271    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:15.676357 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.676280    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:15.676357 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.676288    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:15.676357 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.676297    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0f6ecb-84d8-43f9-add2-2c48115c0fba-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:15.826586 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.826552    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"]
Oct 30 20:54:15.827258 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.827241    4810 cpu_manager.go:414] "containerMap: removing container" podUID="69a2f386-5580-42f4-9066-82a9179735d5" containerName="revoke-keys"
Oct 30 20:54:15.827316 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.827260    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a2f386-5580-42f4-9066-82a9179735d5" containerName="revoke-keys"
Oct 30 20:54:15.827316 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.827271    4810 cpu_manager.go:414] "containerMap: removing container" podUID="b3baa8c3-93c9-4a97-a988-21f4645e361d" containerName="test-curl-7615-065799"
Oct 30 20:54:15.827316 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.827276    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3baa8c3-93c9-4a97-a988-21f4645e361d" containerName="test-curl-7615-065799"
Oct 30 20:54:15.827316 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.827315    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" containerName="istio-proxy"
Oct 30 20:54:15.827441 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.827320    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" containerName="istio-proxy"
Oct 30 20:54:15.827441 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.827433    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a2f386-5580-42f4-9066-82a9179735d5" containerName="revoke-keys"
Oct 30 20:54:15.827500 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.827441    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3baa8c3-93c9-4a97-a988-21f4645e361d" containerName="test-curl-7615-065799"
Oct 30 20:54:15.827500 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.827451    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" containerName="istio-proxy"
Oct 30 20:54:15.832818 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.832784    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:15.838303 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.838279    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"]
Oct 30 20:54:15.907262 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.907228    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"]
Oct 30 20:54:15.913086 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.913056    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-mdq5v"]
Oct 30 20:54:15.982240 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.982194    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:15.982446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.982368    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:15.982446 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:15.982420    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcg76\" (UniqueName: \"kubernetes.io/projected/ca9d891e-817c-4858-b2da-0c361bbfbb7e-kube-api-access-zcg76\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:16.084091 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.083993    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:16.084579 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.084118    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:16.084579 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.084158    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcg76\" (UniqueName: \"kubernetes.io/projected/ca9d891e-817c-4858-b2da-0c361bbfbb7e-kube-api-access-zcg76\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:16.084816 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.084791    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:16.086631 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.086597    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:16.092795 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.092764    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcg76\" (UniqueName: \"kubernetes.io/projected/ca9d891e-817c-4858-b2da-0c361bbfbb7e-kube-api-access-zcg76\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:16.151157 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.151123    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:16.441549 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.441511    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"]
Oct 30 20:54:16.441779 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:54:16.441759    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9d891e_817c_4858_b2da_0c361bbfbb7e.slice/crio-a01d20dd613d03d2e1aaa91e8cbd7c5b7460cb4570831e1e7347033fee99612c WatchSource:0}: Error finding container a01d20dd613d03d2e1aaa91e8cbd7c5b7460cb4570831e1e7347033fee99612c: Status 404 returned error can't find the container with id a01d20dd613d03d2e1aaa91e8cbd7c5b7460cb4570831e1e7347033fee99612c
Oct 30 20:54:16.600907 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.600837    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22" event={"ID":"ca9d891e-817c-4858-b2da-0c361bbfbb7e","Type":"ContainerStarted","Data":"a01d20dd613d03d2e1aaa91e8cbd7c5b7460cb4570831e1e7347033fee99612c"}
Oct 30 20:54:16.603026 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.602493    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7615-065799" podUID="b3baa8c3-93c9-4a97-a988-21f4645e361d" containerName="test-curl-7615-065799" containerID="cri-o://a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c" gracePeriod=2
Oct 30 20:54:16.864765 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:16.864736    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:17.004073 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.004041    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-xks-ca\") pod \"393e6892-57bd-4867-a88a-0ee89441fd04\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") "
Oct 30 20:54:17.004286 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.004090    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-openshift-ca\") pod \"393e6892-57bd-4867-a88a-0ee89441fd04\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") "
Oct 30 20:54:17.004286 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.004137    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393e6892-57bd-4867-a88a-0ee89441fd04-kube-api-access-9qmdl\" (UniqueName: \"kubernetes.io/projected/393e6892-57bd-4867-a88a-0ee89441fd04-kube-api-access-9qmdl\") pod \"393e6892-57bd-4867-a88a-0ee89441fd04\" (UID: \"393e6892-57bd-4867-a88a-0ee89441fd04\") "
Oct 30 20:54:17.004508 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.004485    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-openshift-ca" pod "393e6892-57bd-4867-a88a-0ee89441fd04" (UID: "393e6892-57bd-4867-a88a-0ee89441fd04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:17.004824 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.004798    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:17.006399 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.006369    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393e6892-57bd-4867-a88a-0ee89441fd04-kube-api-access-9qmdl" pod "393e6892-57bd-4867-a88a-0ee89441fd04" (UID: "393e6892-57bd-4867-a88a-0ee89441fd04"). InnerVolumeSpecName "kube-api-access-9qmdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:17.006651 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.006635    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-xks-ca" pod "393e6892-57bd-4867-a88a-0ee89441fd04" (UID: "393e6892-57bd-4867-a88a-0ee89441fd04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:17.106372 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.106338    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/393e6892-57bd-4867-a88a-0ee89441fd04-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:17.106372 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.106367    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qmdl\" (UniqueName: \"kubernetes.io/projected/393e6892-57bd-4867-a88a-0ee89441fd04-kube-api-access-9qmdl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:17.612701 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.612664    4810 generic.go:356] "Generic (PLEG): container finished" podID="ca9d891e-817c-4858-b2da-0c361bbfbb7e" containerID="dd53a81c5ed927d44f9b6eb308ab69d6f8b846b880be07e14df999102642c10e" exitCode=0
Oct 30 20:54:17.612921 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.612750    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22" event={"ID":"ca9d891e-817c-4858-b2da-0c361bbfbb7e","Type":"ContainerDied","Data":"dd53a81c5ed927d44f9b6eb308ab69d6f8b846b880be07e14df999102642c10e"}
Oct 30 20:54:17.615164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.615134    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv" event={"ID":"393e6892-57bd-4867-a88a-0ee89441fd04","Type":"ContainerDied","Data":"0191986d38805e3143fafe72acf9c7f5bcc9a851080ae915f214221b676f5ebd"}
Oct 30 20:54:17.615311 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.615187    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0191986d38805e3143fafe72acf9c7f5bcc9a851080ae915f214221b676f5ebd"
Oct 30 20:54:17.615311 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.615261    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"
Oct 30 20:54:17.662759 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.662690    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0f6ecb-84d8-43f9-add2-2c48115c0fba" path="/var/lib/kubelet/pods/9e0f6ecb-84d8-43f9-add2-2c48115c0fba/volumes"
Oct 30 20:54:17.676092 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:17.676067    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:54:18.835841 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:18.835814    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:18.893370 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:18.893346    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-065799"
Oct 30 20:54:18.937888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:18.937833    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9d891e-817c-4858-b2da-0c361bbfbb7e-kube-api-access-zcg76\" (UniqueName: \"kubernetes.io/projected/ca9d891e-817c-4858-b2da-0c361bbfbb7e-kube-api-access-zcg76\") pod \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") "
Oct 30 20:54:18.938045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:18.938027    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-xks-ca\") pod \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") "
Oct 30 20:54:18.938158 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:18.938141    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-openshift-ca\") pod \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\" (UID: \"ca9d891e-817c-4858-b2da-0c361bbfbb7e\") "
Oct 30 20:54:18.938560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:18.938531    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-openshift-ca" pod "ca9d891e-817c-4858-b2da-0c361bbfbb7e" (UID: "ca9d891e-817c-4858-b2da-0c361bbfbb7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:18.938662 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:18.938629    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:18.940033 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:18.940010    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-xks-ca" pod "ca9d891e-817c-4858-b2da-0c361bbfbb7e" (UID: "ca9d891e-817c-4858-b2da-0c361bbfbb7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:18.940138 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:18.940107    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9d891e-817c-4858-b2da-0c361bbfbb7e-kube-api-access-zcg76" pod "ca9d891e-817c-4858-b2da-0c361bbfbb7e" (UID: "ca9d891e-817c-4858-b2da-0c361bbfbb7e"). InnerVolumeSpecName "kube-api-access-zcg76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:19.040023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.039981    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3baa8c3-93c9-4a97-a988-21f4645e361d-kube-api-access-5plsb\" (UniqueName: \"kubernetes.io/projected/b3baa8c3-93c9-4a97-a988-21f4645e361d-kube-api-access-5plsb\") pod \"b3baa8c3-93c9-4a97-a988-21f4645e361d\" (UID: \"b3baa8c3-93c9-4a97-a988-21f4645e361d\") "
Oct 30 20:54:19.040402 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.040389    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcg76\" (UniqueName: \"kubernetes.io/projected/ca9d891e-817c-4858-b2da-0c361bbfbb7e-kube-api-access-zcg76\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:19.040442 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.040405    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9d891e-817c-4858-b2da-0c361bbfbb7e-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:19.041976 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.041955    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3baa8c3-93c9-4a97-a988-21f4645e361d-kube-api-access-5plsb" pod "b3baa8c3-93c9-4a97-a988-21f4645e361d" (UID: "b3baa8c3-93c9-4a97-a988-21f4645e361d"). InnerVolumeSpecName "kube-api-access-5plsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:19.141328 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.141296    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5plsb\" (UniqueName: \"kubernetes.io/projected/b3baa8c3-93c9-4a97-a988-21f4645e361d-kube-api-access-5plsb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:19.633919 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.633880    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"
Oct 30 20:54:19.634111 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.633921    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22" event={"ID":"ca9d891e-817c-4858-b2da-0c361bbfbb7e","Type":"ContainerDied","Data":"a01d20dd613d03d2e1aaa91e8cbd7c5b7460cb4570831e1e7347033fee99612c"}
Oct 30 20:54:19.634111 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.633983    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a01d20dd613d03d2e1aaa91e8cbd7c5b7460cb4570831e1e7347033fee99612c"
Oct 30 20:54:19.635705 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.635678    4810 generic.go:356] "Generic (PLEG): container finished" podID="b3baa8c3-93c9-4a97-a988-21f4645e361d" containerID="a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c" exitCode=137
Oct 30 20:54:19.635837 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.635717    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-065799"
Oct 30 20:54:19.635837 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.635771    4810 scope.go:122] "RemoveContainer" containerID="a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c"
Oct 30 20:54:19.648237 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.648210    4810 scope.go:122] "RemoveContainer" containerID="a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c"
Oct 30 20:54:19.648611 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:54:19.648585    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c\": container with ID starting with a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c not found: ID does not exist" containerID="a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c"
Oct 30 20:54:19.648717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.648621    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c"} err="failed to get container status \"a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c\": rpc error: code = NotFound desc = could not find container \"a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c\": container with ID starting with a07afcf940d907813f00a9553b49f340b53c07ae77f2baceb25d56b68b38fc9c not found: ID does not exist"
Oct 30 20:54:19.654238 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:54:19.654208    4810 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7615-065799\" is forbidden: User \"system:node:ip-10-0-128-161.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-161.us-east-2.compute.internal' and this object" podUID="b3baa8c3-93c9-4a97-a988-21f4645e361d" pod="opendatahub/test-curl-7615-065799"
Oct 30 20:54:19.656040 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:54:19.656017    4810 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7615-065799\" is forbidden: User \"system:node:ip-10-0-128-161.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-161.us-east-2.compute.internal' and this object" podUID="b3baa8c3-93c9-4a97-a988-21f4645e361d" pod="opendatahub/test-curl-7615-065799"
Oct 30 20:54:19.662502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:19.662481    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3baa8c3-93c9-4a97-a988-21f4645e361d" path="/var/lib/kubelet/pods/b3baa8c3-93c9-4a97-a988-21f4645e361d/volumes"
Oct 30 20:54:20.602392 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:20.602359    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"]
Oct 30 20:54:20.606929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:20.606901    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-95a684-zbbtl"]
Oct 30 20:54:21.669669 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:21.669631    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91098eef-5a42-4d57-a9a6-f805bc2c6143" path="/var/lib/kubelet/pods/91098eef-5a42-4d57-a9a6-f805bc2c6143/volumes"
Oct 30 20:54:24.225585 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:24.225550    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"]
Oct 30 20:54:24.229348 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:24.229319    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1982ff-rq4wr"]
Oct 30 20:54:25.156757 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:54:25.156685    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:54:25.662314 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:25.662287    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f56755f-347b-47d9-be99-5b72184fdbe8" path="/var/lib/kubelet/pods/7f56755f-347b-47d9-be99-5b72184fdbe8/volumes"
Oct 30 20:54:28.036114 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:28.036073    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"]
Oct 30 20:54:28.042433 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:28.042403    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68a3758-5h5jj"]
Oct 30 20:54:29.053561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.053521    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"]
Oct 30 20:54:29.057473 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.057439    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4adaa4a7-2hk8z"]
Oct 30 20:54:29.164969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.164938    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"]
Oct 30 20:54:29.165618 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.165602    4810 cpu_manager.go:414] "containerMap: removing container" podUID="393e6892-57bd-4867-a88a-0ee89441fd04" containerName="revoke-keys"
Oct 30 20:54:29.165669 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.165619    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393e6892-57bd-4867-a88a-0ee89441fd04" containerName="revoke-keys"
Oct 30 20:54:29.165669 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.165647    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ca9d891e-817c-4858-b2da-0c361bbfbb7e" containerName="revoke-keys"
Oct 30 20:54:29.165669 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.165652    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9d891e-817c-4858-b2da-0c361bbfbb7e" containerName="revoke-keys"
Oct 30 20:54:29.165772 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.165762    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="393e6892-57bd-4867-a88a-0ee89441fd04" containerName="revoke-keys"
Oct 30 20:54:29.165805 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.165776    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9d891e-817c-4858-b2da-0c361bbfbb7e" containerName="revoke-keys"
Oct 30 20:54:29.170704 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.170687    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.175011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.174988    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:54:29.175346 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.175320    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:54:29.180353 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.180331    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"]
Oct 30 20:54:29.245037 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.244995    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84f901-wn76v\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.245175 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.245157    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84f901-wn76v\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.245268 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.245246    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlcw\" (UniqueName: \"kubernetes.io/projected/3316c8b0-e213-4c24-8274-3a188a3ece1b-kube-api-access-fwlcw\") pod \"maas-api-revoke-keys-e2e-api-b-84f901-wn76v\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.346778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.346693    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84f901-wn76v\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.346945 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.346813    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84f901-wn76v\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.346945 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.346891    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlcw\" (UniqueName: \"kubernetes.io/projected/3316c8b0-e213-4c24-8274-3a188a3ece1b-kube-api-access-fwlcw\") pod \"maas-api-revoke-keys-e2e-api-b-84f901-wn76v\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.347373 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.347349    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84f901-wn76v\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.349132 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.349109    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84f901-wn76v\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.363961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.363937    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlcw\" (UniqueName: \"kubernetes.io/projected/3316c8b0-e213-4c24-8274-3a188a3ece1b-kube-api-access-fwlcw\") pod \"maas-api-revoke-keys-e2e-api-b-84f901-wn76v\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.490175 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.490151    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:29.662216 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.662188    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b0e915-e59e-4f23-b4e4-c68bcb4e48e5" path="/var/lib/kubelet/pods/41b0e915-e59e-4f23-b4e4-c68bcb4e48e5/volumes"
Oct 30 20:54:29.662633 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.662619    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6aa8b9c-9151-4b39-af65-541dbba0ccd7" path="/var/lib/kubelet/pods/f6aa8b9c-9151-4b39-af65-541dbba0ccd7/volumes"
Oct 30 20:54:29.684693 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.684659    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"]
Oct 30 20:54:29.689558 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.689538    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:29.708121 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.708102    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"]
Oct 30 20:54:29.780724 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.780694    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"]
Oct 30 20:54:29.781626 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:54:29.781601    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3316c8b0_e213_4c24_8274_3a188a3ece1b.slice/crio-cbda854116dfe20e9829bcd1d239b8f8ed0730aa84aa2d748caf792c8e58ec04 WatchSource:0}: Error finding container cbda854116dfe20e9829bcd1d239b8f8ed0730aa84aa2d748caf792c8e58ec04: Status 404 returned error can't find the container with id cbda854116dfe20e9829bcd1d239b8f8ed0730aa84aa2d748caf792c8e58ec04
Oct 30 20:54:29.852952 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.852923    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe1ea85-zxrk2\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:29.853159 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.853137    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe1ea85-zxrk2\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:29.853215 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.853204    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksf9\" (UniqueName: \"kubernetes.io/projected/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-kube-api-access-rksf9\") pod \"maas-api-revoke-sub-0fe1ea85-zxrk2\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:29.954415 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.954339    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe1ea85-zxrk2\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:29.954565 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.954550    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe1ea85-zxrk2\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:29.954644 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.954629    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksf9\" (UniqueName: \"kubernetes.io/projected/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-kube-api-access-rksf9\") pod \"maas-api-revoke-sub-0fe1ea85-zxrk2\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:29.955211 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.955188    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fe1ea85-zxrk2\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:29.956589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.956573    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fe1ea85-zxrk2\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:29.967801 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:29.967778    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksf9\" (UniqueName: \"kubernetes.io/projected/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-kube-api-access-rksf9\") pod \"maas-api-revoke-sub-0fe1ea85-zxrk2\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:30.006783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:30.004794    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:30.305892 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:30.305853    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"]
Oct 30 20:54:30.306297 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:54:30.306275    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7be06b_7f5a_43e2_9e0c_2b9a8ec46a33.slice/crio-a4e943f8003c5ae7c8b2cef3ea24781f756899088c25cbda48c0d16d6177917b WatchSource:0}: Error finding container a4e943f8003c5ae7c8b2cef3ea24781f756899088c25cbda48c0d16d6177917b: Status 404 returned error can't find the container with id a4e943f8003c5ae7c8b2cef3ea24781f756899088c25cbda48c0d16d6177917b
Oct 30 20:54:30.767046 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:30.767012    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2" event={"ID":"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33","Type":"ContainerStarted","Data":"9cf1a65535916ae54df5294f6c14e47a5ed2ee2334023deac3acbaac5b250859"}
Oct 30 20:54:30.767193 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:30.767075    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2" event={"ID":"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33","Type":"ContainerStarted","Data":"a4e943f8003c5ae7c8b2cef3ea24781f756899088c25cbda48c0d16d6177917b"}
Oct 30 20:54:30.768619 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:30.768590    4810 generic.go:356] "Generic (PLEG): container finished" podID="3316c8b0-e213-4c24-8274-3a188a3ece1b" containerID="a4262b7d9c07c5213f29ca6c36eaa2fa35aab99efe8474693a96aa48babe0b7d" exitCode=0
Oct 30 20:54:30.768722 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:30.768673    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v" event={"ID":"3316c8b0-e213-4c24-8274-3a188a3ece1b","Type":"ContainerDied","Data":"a4262b7d9c07c5213f29ca6c36eaa2fa35aab99efe8474693a96aa48babe0b7d"}
Oct 30 20:54:30.768777 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:30.768719    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v" event={"ID":"3316c8b0-e213-4c24-8274-3a188a3ece1b","Type":"ContainerStarted","Data":"cbda854116dfe20e9829bcd1d239b8f8ed0730aa84aa2d748caf792c8e58ec04"}
Oct 30 20:54:30.790478 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:30.790432    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2" podStartSLOduration=1.790419713 podStartE2EDuration="1.790419713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:30.784667766 +0000 UTC m=+1037.276484312" watchObservedRunningTime="2026-10-30 20:54:30.790419713 +0000 UTC m=+1037.282236258"
Oct 30 20:54:31.779159 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:31.779125    4810 generic.go:356] "Generic (PLEG): container finished" podID="bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33" containerID="9cf1a65535916ae54df5294f6c14e47a5ed2ee2334023deac3acbaac5b250859" exitCode=0
Oct 30 20:54:31.779629 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:31.779209    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2" event={"ID":"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33","Type":"ContainerDied","Data":"9cf1a65535916ae54df5294f6c14e47a5ed2ee2334023deac3acbaac5b250859"}
Oct 30 20:54:32.010194 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.010168    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:32.098489 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.098415    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-openshift-ca\") pod \"3316c8b0-e213-4c24-8274-3a188a3ece1b\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") "
Oct 30 20:54:32.098489 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.098448    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3316c8b0-e213-4c24-8274-3a188a3ece1b-kube-api-access-fwlcw\" (UniqueName: \"kubernetes.io/projected/3316c8b0-e213-4c24-8274-3a188a3ece1b-kube-api-access-fwlcw\") pod \"3316c8b0-e213-4c24-8274-3a188a3ece1b\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") "
Oct 30 20:54:32.098489 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.098471    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-xks-ca\") pod \"3316c8b0-e213-4c24-8274-3a188a3ece1b\" (UID: \"3316c8b0-e213-4c24-8274-3a188a3ece1b\") "
Oct 30 20:54:32.098806 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.098765    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-openshift-ca" pod "3316c8b0-e213-4c24-8274-3a188a3ece1b" (UID: "3316c8b0-e213-4c24-8274-3a188a3ece1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:32.099039 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.099021    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:32.100604 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.100575    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-xks-ca" pod "3316c8b0-e213-4c24-8274-3a188a3ece1b" (UID: "3316c8b0-e213-4c24-8274-3a188a3ece1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:32.100698 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.100651    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3316c8b0-e213-4c24-8274-3a188a3ece1b-kube-api-access-fwlcw" pod "3316c8b0-e213-4c24-8274-3a188a3ece1b" (UID: "3316c8b0-e213-4c24-8274-3a188a3ece1b"). InnerVolumeSpecName "kube-api-access-fwlcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:32.200081 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.200047    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwlcw\" (UniqueName: \"kubernetes.io/projected/3316c8b0-e213-4c24-8274-3a188a3ece1b-kube-api-access-fwlcw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:32.200081 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.200073    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3316c8b0-e213-4c24-8274-3a188a3ece1b-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:32.806097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.806066    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"
Oct 30 20:54:32.806994 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.806956    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v" event={"ID":"3316c8b0-e213-4c24-8274-3a188a3ece1b","Type":"ContainerDied","Data":"cbda854116dfe20e9829bcd1d239b8f8ed0730aa84aa2d748caf792c8e58ec04"}
Oct 30 20:54:32.807078 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:32.807041    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbda854116dfe20e9829bcd1d239b8f8ed0730aa84aa2d748caf792c8e58ec04"
Oct 30 20:54:33.133924 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.132624    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"]
Oct 30 20:54:33.137738 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.137709    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49990a-f7z8n"]
Oct 30 20:54:33.199778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.199753    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:33.221398 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.221364    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-openshift-ca\") pod \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") "
Oct 30 20:54:33.221564 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.221497    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-kube-api-access-rksf9\" (UniqueName: \"kubernetes.io/projected/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-kube-api-access-rksf9\") pod \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") "
Oct 30 20:54:33.221638 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.221614    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-xks-ca\") pod \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\" (UID: \"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33\") "
Oct 30 20:54:33.222108 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.222082    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-openshift-ca" pod "bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33" (UID: "bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:33.228899 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.227146    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-xks-ca" pod "bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33" (UID: "bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:33.228899 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.227170    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-kube-api-access-rksf9" pod "bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33" (UID: "bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33"). InnerVolumeSpecName "kube-api-access-rksf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:33.323881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.323834    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:33.323881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.323886    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rksf9\" (UniqueName: \"kubernetes.io/projected/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-kube-api-access-rksf9\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:33.324135 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.323901    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:33.398756 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.398685    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"]
Oct 30 20:54:33.399310 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.399296    4810 cpu_manager.go:414] "containerMap: removing container" podUID="bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33" containerName="revoke-keys"
Oct 30 20:54:33.399364 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.399312    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33" containerName="revoke-keys"
Oct 30 20:54:33.399364 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.399325    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3316c8b0-e213-4c24-8274-3a188a3ece1b" containerName="revoke-keys"
Oct 30 20:54:33.399364 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.399330    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3316c8b0-e213-4c24-8274-3a188a3ece1b" containerName="revoke-keys"
Oct 30 20:54:33.399512 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.399492    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33" containerName="revoke-keys"
Oct 30 20:54:33.399512 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.399513    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3316c8b0-e213-4c24-8274-3a188a3ece1b" containerName="revoke-keys"
Oct 30 20:54:33.404001 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.403981    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.415198 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.415175    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"]
Oct 30 20:54:33.424756 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.424720    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0333a9-g88nh\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.424925 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.424820    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0333a9-g88nh\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.425030 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.425013    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jjls\" (UniqueName: \"kubernetes.io/projected/c39edec2-acb0-4b82-9599-e5936dceafe0-kube-api-access-9jjls\") pod \"maas-api-revoke-keys-e2e-api-b-0333a9-g88nh\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.526757 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.526720    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jjls\" (UniqueName: \"kubernetes.io/projected/c39edec2-acb0-4b82-9599-e5936dceafe0-kube-api-access-9jjls\") pod \"maas-api-revoke-keys-e2e-api-b-0333a9-g88nh\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.526939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.526822    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0333a9-g88nh\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.526939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.526930    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0333a9-g88nh\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.527755 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.527727    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0333a9-g88nh\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.529511 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.529488    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0333a9-g88nh\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.536620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.536601    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jjls\" (UniqueName: \"kubernetes.io/projected/c39edec2-acb0-4b82-9599-e5936dceafe0-kube-api-access-9jjls\") pod \"maas-api-revoke-keys-e2e-api-b-0333a9-g88nh\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.664778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.664692    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a2f386-5580-42f4-9066-82a9179735d5" path="/var/lib/kubelet/pods/69a2f386-5580-42f4-9066-82a9179735d5/volumes"
Oct 30 20:54:33.723062 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.723036    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:33.819165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.819139    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"
Oct 30 20:54:33.819609 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.819159    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2" event={"ID":"bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33","Type":"ContainerDied","Data":"a4e943f8003c5ae7c8b2cef3ea24781f756899088c25cbda48c0d16d6177917b"}
Oct 30 20:54:33.819609 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:33.819215    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e943f8003c5ae7c8b2cef3ea24781f756899088c25cbda48c0d16d6177917b"
Oct 30 20:54:34.025564 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:34.025526    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"]
Oct 30 20:54:34.025742 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:54:34.025723    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39edec2_acb0_4b82_9599_e5936dceafe0.slice/crio-48d59e5bb8c31235b9996c56fb46c9ba91c9cf2c766ed7fb845939552035db9c WatchSource:0}: Error finding container 48d59e5bb8c31235b9996c56fb46c9ba91c9cf2c766ed7fb845939552035db9c: Status 404 returned error can't find the container with id 48d59e5bb8c31235b9996c56fb46c9ba91c9cf2c766ed7fb845939552035db9c
Oct 30 20:54:34.830953 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:34.830845    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh" event={"ID":"c39edec2-acb0-4b82-9599-e5936dceafe0","Type":"ContainerStarted","Data":"1b156f84fe8dc4b209ab4990b805f8dce5294d73459534dde6522998876b372f"}
Oct 30 20:54:34.830953 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:34.830926    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh" event={"ID":"c39edec2-acb0-4b82-9599-e5936dceafe0","Type":"ContainerStarted","Data":"48d59e5bb8c31235b9996c56fb46c9ba91c9cf2c766ed7fb845939552035db9c"}
Oct 30 20:54:34.856847 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:34.856773    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh" podStartSLOduration=1.856756163 podStartE2EDuration="1.856756163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:34.84969928 +0000 UTC m=+1041.341515826" watchObservedRunningTime="2026-10-30 20:54:34.856756163 +0000 UTC m=+1041.348572710"
Oct 30 20:54:35.840796 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:35.840764    4810 generic.go:356] "Generic (PLEG): container finished" podID="c39edec2-acb0-4b82-9599-e5936dceafe0" containerID="1b156f84fe8dc4b209ab4990b805f8dce5294d73459534dde6522998876b372f" exitCode=0
Oct 30 20:54:35.841232 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:35.840869    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh" event={"ID":"c39edec2-acb0-4b82-9599-e5936dceafe0","Type":"ContainerDied","Data":"1b156f84fe8dc4b209ab4990b805f8dce5294d73459534dde6522998876b372f"}
Oct 30 20:54:37.026544 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.026516    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:37.086398 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.086345    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39edec2-acb0-4b82-9599-e5936dceafe0-kube-api-access-9jjls\" (UniqueName: \"kubernetes.io/projected/c39edec2-acb0-4b82-9599-e5936dceafe0-kube-api-access-9jjls\") pod \"c39edec2-acb0-4b82-9599-e5936dceafe0\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") "
Oct 30 20:54:37.087120 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.087094    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-openshift-ca\") pod \"c39edec2-acb0-4b82-9599-e5936dceafe0\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") "
Oct 30 20:54:37.087340 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.087321    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-xks-ca\") pod \"c39edec2-acb0-4b82-9599-e5936dceafe0\" (UID: \"c39edec2-acb0-4b82-9599-e5936dceafe0\") "
Oct 30 20:54:37.088849 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.088806    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-openshift-ca" pod "c39edec2-acb0-4b82-9599-e5936dceafe0" (UID: "c39edec2-acb0-4b82-9599-e5936dceafe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:37.109773 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.109152    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39edec2-acb0-4b82-9599-e5936dceafe0-kube-api-access-9jjls" pod "c39edec2-acb0-4b82-9599-e5936dceafe0" (UID: "c39edec2-acb0-4b82-9599-e5936dceafe0"). InnerVolumeSpecName "kube-api-access-9jjls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:37.120540 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.120479    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-xks-ca" pod "c39edec2-acb0-4b82-9599-e5936dceafe0" (UID: "c39edec2-acb0-4b82-9599-e5936dceafe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:37.189372 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.189337    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jjls\" (UniqueName: \"kubernetes.io/projected/c39edec2-acb0-4b82-9599-e5936dceafe0-kube-api-access-9jjls\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:37.189372 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.189371    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:37.189600 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.189390    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39edec2-acb0-4b82-9599-e5936dceafe0-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:37.858031 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.857998    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"
Oct 30 20:54:37.858031 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.858010    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh" event={"ID":"c39edec2-acb0-4b82-9599-e5936dceafe0","Type":"ContainerDied","Data":"48d59e5bb8c31235b9996c56fb46c9ba91c9cf2c766ed7fb845939552035db9c"}
Oct 30 20:54:37.858250 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:37.858054    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48d59e5bb8c31235b9996c56fb46c9ba91c9cf2c766ed7fb845939552035db9c"
Oct 30 20:54:39.301572 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:39.301535    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"]
Oct 30 20:54:39.335903 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:39.332142    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"]
Oct 30 20:54:40.026534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:40.026494    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"]
Oct 30 20:54:40.031734 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:40.031705    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e14b6ff-4zdzj"]
Oct 30 20:54:41.662912 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:41.662852    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df985564-7f1b-4772-84c7-f3fb9486b6dd" path="/var/lib/kubelet/pods/df985564-7f1b-4772-84c7-f3fb9486b6dd/volumes"
Oct 30 20:54:42.584224 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:42.584191    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:54:43.592167 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:43.592122    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"]
Oct 30 20:54:43.601488 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:43.601458    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"]
Oct 30 20:54:44.301877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:44.301824    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw" podUID="0149b3ca-ab00-4c5a-a8a8-549042da1fd8" containerName="payload-processing" containerID="cri-o://083dc7192fb6c0100b68b983527ddc4e8e641b89690fac3db1df720243da9d34" gracePeriod=25
Oct 30 20:54:44.332791 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:44.332661    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r" podUID="1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6" containerName="payload-pre-processing" containerID="cri-o://be2fe866a439d0640f92cf2e45f6474d4e22261535b28c93f0ebbd411037aaf1" gracePeriod=25
Oct 30 20:54:45.565005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.564333    4810 generic.go:356] "Generic (PLEG): container finished" podID="1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6" containerID="be2fe866a439d0640f92cf2e45f6474d4e22261535b28c93f0ebbd411037aaf1" exitCode=0
Oct 30 20:54:45.565005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.564481    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r" event={"ID":"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6","Type":"ContainerDied","Data":"be2fe866a439d0640f92cf2e45f6474d4e22261535b28c93f0ebbd411037aaf1"}
Oct 30 20:54:45.616470 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.616138    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:54:45.617609 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.617580    4810 generic.go:356] "Generic (PLEG): container finished" podID="0149b3ca-ab00-4c5a-a8a8-549042da1fd8" containerID="083dc7192fb6c0100b68b983527ddc4e8e641b89690fac3db1df720243da9d34" exitCode=0
Oct 30 20:54:45.620102 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.617958    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw" event={"ID":"0149b3ca-ab00-4c5a-a8a8-549042da1fd8","Type":"ContainerDied","Data":"083dc7192fb6c0100b68b983527ddc4e8e641b89690fac3db1df720243da9d34"}
Oct 30 20:54:45.620299 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.620284    4810 scope.go:122] "RemoveContainer" containerID="083dc7192fb6c0100b68b983527ddc4e8e641b89690fac3db1df720243da9d34"
Oct 30 20:54:45.621659 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.621601    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:54:45.705949 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.699332    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-plugins-config-volume\") pod \"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6\" (UID: \"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6\") "
Oct 30 20:54:45.705949 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.699399    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-kube-api-access-g4k8m\" (UniqueName: \"kubernetes.io/projected/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-kube-api-access-g4k8m\") pod \"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6\" (UID: \"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6\") "
Oct 30 20:54:45.705949 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.699603    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-kube-api-access-lfxr7\" (UniqueName: \"kubernetes.io/projected/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-kube-api-access-lfxr7\") pod \"0149b3ca-ab00-4c5a-a8a8-549042da1fd8\" (UID: \"0149b3ca-ab00-4c5a-a8a8-549042da1fd8\") "
Oct 30 20:54:45.705949 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.699720    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-plugins-config-volume\") pod \"0149b3ca-ab00-4c5a-a8a8-549042da1fd8\" (UID: \"0149b3ca-ab00-4c5a-a8a8-549042da1fd8\") "
Oct 30 20:54:45.705949 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.701253    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-plugins-config-volume" pod "1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6" (UID: "1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:45.705949 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.701675    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-plugins-config-volume" pod "0149b3ca-ab00-4c5a-a8a8-549042da1fd8" (UID: "0149b3ca-ab00-4c5a-a8a8-549042da1fd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:45.753114 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.753065    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-kube-api-access-lfxr7" pod "0149b3ca-ab00-4c5a-a8a8-549042da1fd8" (UID: "0149b3ca-ab00-4c5a-a8a8-549042da1fd8"). InnerVolumeSpecName "kube-api-access-lfxr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:45.754009 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.753961    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-kube-api-access-g4k8m" pod "1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6" (UID: "1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6"). InnerVolumeSpecName "kube-api-access-g4k8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:45.806878 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.802157    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.806878 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.802188    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.806878 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.802207    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4k8m\" (UniqueName: \"kubernetes.io/projected/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6-kube-api-access-g4k8m\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.806878 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:45.802226    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfxr7\" (UniqueName: \"kubernetes.io/projected/0149b3ca-ab00-4c5a-a8a8-549042da1fd8-kube-api-access-lfxr7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:46.629355 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:46.629328    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"
Oct 30 20:54:46.629792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:46.629376    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r" event={"ID":"1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6","Type":"ContainerDied","Data":"b0fa852b4b0ce46906bc08e95d7dd1b491470ec1ac4a2726793c18d30daadeb5"}
Oct 30 20:54:46.629792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:46.629441    4810 scope.go:122] "RemoveContainer" containerID="be2fe866a439d0640f92cf2e45f6474d4e22261535b28c93f0ebbd411037aaf1"
Oct 30 20:54:46.630737 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:46.630609    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"
Oct 30 20:54:46.630737 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:46.630652    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw" event={"ID":"0149b3ca-ab00-4c5a-a8a8-549042da1fd8","Type":"ContainerDied","Data":"34a33f7d911e2b7b2a79009074a8d570edbbaf1fe2bb257f8b15d57c334a42c8"}
Oct 30 20:54:46.652843 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:46.652821    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"]
Oct 30 20:54:46.662467 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:46.662445    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-84f901-ff44cf6df-9m2nw"]
Oct 30 20:54:46.680685 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:46.680660    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"]
Oct 30 20:54:46.684329 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:46.684307    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84f901-56d5f4f4c9-9tq8r"]
Oct 30 20:54:47.661941 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:47.661906    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0149b3ca-ab00-4c5a-a8a8-549042da1fd8" path="/var/lib/kubelet/pods/0149b3ca-ab00-4c5a-a8a8-549042da1fd8/volumes"
Oct 30 20:54:47.662327 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:47.662313    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6" path="/var/lib/kubelet/pods/1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6/volumes"
Oct 30 20:54:48.592586 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:48.592539    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5" podUID="b9c0902f-9319-477d-a19b-6a0aecd0a1dd" containerName="payload-processing" containerID="cri-o://76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf" gracePeriod=25
Oct 30 20:54:48.602193 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:48.602163    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6" podUID="f2c37f9e-5a66-46e1-b83d-420a84dd5e4a" containerName="payload-pre-processing" containerID="cri-o://803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b" gracePeriod=25
Oct 30 20:54:48.937334 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:48.937308    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:54:48.971918 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:48.971888    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:54:49.052804 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.052767    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-kube-api-access-5p45g\" (UniqueName: \"kubernetes.io/projected/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-kube-api-access-5p45g\") pod \"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a\" (UID: \"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a\") "
Oct 30 20:54:49.053023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.052887    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-kube-api-access-xh48k\" (UniqueName: \"kubernetes.io/projected/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-kube-api-access-xh48k\") pod \"b9c0902f-9319-477d-a19b-6a0aecd0a1dd\" (UID: \"b9c0902f-9319-477d-a19b-6a0aecd0a1dd\") "
Oct 30 20:54:49.053023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.052948    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-plugins-config-volume\") pod \"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a\" (UID: \"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a\") "
Oct 30 20:54:49.053145 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.053021    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-plugins-config-volume\") pod \"b9c0902f-9319-477d-a19b-6a0aecd0a1dd\" (UID: \"b9c0902f-9319-477d-a19b-6a0aecd0a1dd\") "
Oct 30 20:54:49.053383 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.053354    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-plugins-config-volume" pod "b9c0902f-9319-477d-a19b-6a0aecd0a1dd" (UID: "b9c0902f-9319-477d-a19b-6a0aecd0a1dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:49.053451 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.053359    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-plugins-config-volume" pod "f2c37f9e-5a66-46e1-b83d-420a84dd5e4a" (UID: "f2c37f9e-5a66-46e1-b83d-420a84dd5e4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:49.053509 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.053494    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:49.053549 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.053514    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:49.054832 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.054814    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-kube-api-access-5p45g" pod "f2c37f9e-5a66-46e1-b83d-420a84dd5e4a" (UID: "f2c37f9e-5a66-46e1-b83d-420a84dd5e4a"). InnerVolumeSpecName "kube-api-access-5p45g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:49.054928 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.054912    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-kube-api-access-xh48k" pod "b9c0902f-9319-477d-a19b-6a0aecd0a1dd" (UID: "b9c0902f-9319-477d-a19b-6a0aecd0a1dd"). InnerVolumeSpecName "kube-api-access-xh48k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:49.154480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.154450    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p45g\" (UniqueName: \"kubernetes.io/projected/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a-kube-api-access-5p45g\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:49.154480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.154477    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh48k\" (UniqueName: \"kubernetes.io/projected/b9c0902f-9319-477d-a19b-6a0aecd0a1dd-kube-api-access-xh48k\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:54:49.710502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.710469    4810 generic.go:356] "Generic (PLEG): container finished" podID="f2c37f9e-5a66-46e1-b83d-420a84dd5e4a" containerID="803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b" exitCode=0
Oct 30 20:54:49.710836 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.710819    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"
Oct 30 20:54:49.712344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.712320    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6" event={"ID":"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a","Type":"ContainerDied","Data":"803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b"}
Oct 30 20:54:49.712947 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.712924    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6" event={"ID":"f2c37f9e-5a66-46e1-b83d-420a84dd5e4a","Type":"ContainerDied","Data":"2a15f58c82186b3cafe1c35baacb7ac64eb07ab4fb39ee930fbdfe3840c2e1f9"}
Oct 30 20:54:49.713091 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.713078    4810 scope.go:122] "RemoveContainer" containerID="803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b"
Oct 30 20:54:49.714456 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.714306    4810 generic.go:356] "Generic (PLEG): container finished" podID="b9c0902f-9319-477d-a19b-6a0aecd0a1dd" containerID="76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf" exitCode=0
Oct 30 20:54:49.715099 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.715078    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5" event={"ID":"b9c0902f-9319-477d-a19b-6a0aecd0a1dd","Type":"ContainerDied","Data":"76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf"}
Oct 30 20:54:49.715247 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.715232    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5" event={"ID":"b9c0902f-9319-477d-a19b-6a0aecd0a1dd","Type":"ContainerDied","Data":"92063ac42c374d2c10f2dfe57abc8482f6dfe6bbad46701a01c97993e9e3813f"}
Oct 30 20:54:49.715400 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.715387    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"
Oct 30 20:54:49.747230 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.747128    4810 scope.go:122] "RemoveContainer" containerID="803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b"
Oct 30 20:54:49.748741 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:54:49.748714    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b\": container with ID starting with 803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b not found: ID does not exist" containerID="803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b"
Oct 30 20:54:49.748910 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.748752    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b"} err="failed to get container status \"803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b\": rpc error: code = NotFound desc = could not find container \"803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b\": container with ID starting with 803809f83865cb1898e9208da034f50ee691a841fced499d5b6421bf7ad9586b not found: ID does not exist"
Oct 30 20:54:49.748910 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.748780    4810 scope.go:122] "RemoveContainer" containerID="76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf"
Oct 30 20:54:49.761205 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.761181    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"]
Oct 30 20:54:49.772947 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.772882    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0333a9-9449bf987-tq4h5"]
Oct 30 20:54:49.784424 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.783968    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"]
Oct 30 20:54:49.798828 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.798805    4810 scope.go:122] "RemoveContainer" containerID="76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf"
Oct 30 20:54:49.799613 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:54:49.799519    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf\": container with ID starting with 76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf not found: ID does not exist" containerID="76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf"
Oct 30 20:54:49.799613 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.799561    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf"} err="failed to get container status \"76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf\": rpc error: code = NotFound desc = could not find container \"76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf\": container with ID starting with 76ee2145865e2d1efeb515baed180cdc4b76326c4a78348d32d92268b00f20cf not found: ID does not exist"
Oct 30 20:54:49.802612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.802589    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0333a9-7454578bbc-g76f6"]
Oct 30 20:54:49.978648 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.978616    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"]
Oct 30 20:54:49.982432 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:49.982403    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-q6l22"]
Oct 30 20:54:51.662370 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:51.662339    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c0902f-9319-477d-a19b-6a0aecd0a1dd" path="/var/lib/kubelet/pods/b9c0902f-9319-477d-a19b-6a0aecd0a1dd/volumes"
Oct 30 20:54:51.662945 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:51.662927    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca9d891e-817c-4858-b2da-0c361bbfbb7e" path="/var/lib/kubelet/pods/ca9d891e-817c-4858-b2da-0c361bbfbb7e/volumes"
Oct 30 20:54:51.663350 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:51.663331    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c37f9e-5a66-46e1-b83d-420a84dd5e4a" path="/var/lib/kubelet/pods/f2c37f9e-5a66-46e1-b83d-420a84dd5e4a/volumes"
Oct 30 20:54:55.157913 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:54:55.157875    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:54:55.686230 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:55.686188    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"]
Oct 30 20:54:55.689834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:55.689809    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0333a9-g88nh"]
Oct 30 20:54:56.203763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:56.203714    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"]
Oct 30 20:54:56.209922 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:56.209890    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84f901-wn76v"]
Oct 30 20:54:57.662564 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:57.662533    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3316c8b0-e213-4c24-8274-3a188a3ece1b" path="/var/lib/kubelet/pods/3316c8b0-e213-4c24-8274-3a188a3ece1b/volumes"
Oct 30 20:54:57.663052 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:54:57.663038    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39edec2-acb0-4b82-9599-e5936dceafe0" path="/var/lib/kubelet/pods/c39edec2-acb0-4b82-9599-e5936dceafe0/volumes"
Oct 30 20:55:05.273838 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:05.273807    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:55:06.468330 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.468293    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"]
Oct 30 20:55:06.469197 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469177    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f2c37f9e-5a66-46e1-b83d-420a84dd5e4a" containerName="payload-pre-processing"
Oct 30 20:55:06.469263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469200    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c37f9e-5a66-46e1-b83d-420a84dd5e4a" containerName="payload-pre-processing"
Oct 30 20:55:06.469263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469230    4810 cpu_manager.go:414] "containerMap: removing container" podUID="0149b3ca-ab00-4c5a-a8a8-549042da1fd8" containerName="payload-processing"
Oct 30 20:55:06.469263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469238    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0149b3ca-ab00-4c5a-a8a8-549042da1fd8" containerName="payload-processing"
Oct 30 20:55:06.469263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469257    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6" containerName="payload-pre-processing"
Oct 30 20:55:06.469393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469265    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6" containerName="payload-pre-processing"
Oct 30 20:55:06.469393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469285    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c39edec2-acb0-4b82-9599-e5936dceafe0" containerName="revoke-keys"
Oct 30 20:55:06.469393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469294    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39edec2-acb0-4b82-9599-e5936dceafe0" containerName="revoke-keys"
Oct 30 20:55:06.469393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469321    4810 cpu_manager.go:414] "containerMap: removing container" podUID="b9c0902f-9319-477d-a19b-6a0aecd0a1dd" containerName="payload-processing"
Oct 30 20:55:06.469393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469328    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c0902f-9319-477d-a19b-6a0aecd0a1dd" containerName="payload-processing"
Oct 30 20:55:06.469535 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469490    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b15a992-c2e3-4cd0-a9f3-a6db6ed2e1d6" containerName="payload-pre-processing"
Oct 30 20:55:06.469535 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469506    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39edec2-acb0-4b82-9599-e5936dceafe0" containerName="revoke-keys"
Oct 30 20:55:06.469535 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469521    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c0902f-9319-477d-a19b-6a0aecd0a1dd" containerName="payload-processing"
Oct 30 20:55:06.469618 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469542    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c37f9e-5a66-46e1-b83d-420a84dd5e4a" containerName="payload-pre-processing"
Oct 30 20:55:06.469618 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.469557    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="0149b3ca-ab00-4c5a-a8a8-549042da1fd8" containerName="payload-processing"
Oct 30 20:55:06.473883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.473848    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:06.479745 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.479725    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:55:06.480090 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.480069    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:55:06.482395 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.482372    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"]
Oct 30 20:55:06.638837 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.638798    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322586cd-nlw89\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:06.639050 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.638955    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322586cd-nlw89\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:06.639193 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.639172    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9fj5\" (UniqueName: \"kubernetes.io/projected/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-kube-api-access-h9fj5\") pod \"maas-api-revoke-sub-322586cd-nlw89\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:06.741154 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.741068    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322586cd-nlw89\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:06.741296 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.741169    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322586cd-nlw89\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:06.741389 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.741374    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fj5\" (UniqueName: \"kubernetes.io/projected/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-kube-api-access-h9fj5\") pod \"maas-api-revoke-sub-322586cd-nlw89\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:06.741892 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.741836    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322586cd-nlw89\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:06.744082 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.744060    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322586cd-nlw89\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:06.750340 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.750316    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fj5\" (UniqueName: \"kubernetes.io/projected/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-kube-api-access-h9fj5\") pod \"maas-api-revoke-sub-322586cd-nlw89\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:06.793797 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:06.793771    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:07.154926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:07.153379    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"]
Oct 30 20:55:07.157531 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:55:07.157500    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1124a4d8_de7b_45f0_8ffc_fd7974ea2117.slice/crio-5d173685b44700ea7449ca41100bc047eaf9e80eb3332f3f8823d1ec507e820c WatchSource:0}: Error finding container 5d173685b44700ea7449ca41100bc047eaf9e80eb3332f3f8823d1ec507e820c: Status 404 returned error can't find the container with id 5d173685b44700ea7449ca41100bc047eaf9e80eb3332f3f8823d1ec507e820c
Oct 30 20:55:07.933930 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:07.933554    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89" event={"ID":"1124a4d8-de7b-45f0-8ffc-fd7974ea2117","Type":"ContainerStarted","Data":"69ced85f2764d73ed2142bc6a3d404909d4a83579f790febba277c9aa339d345"}
Oct 30 20:55:07.933930 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:07.933669    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89" event={"ID":"1124a4d8-de7b-45f0-8ffc-fd7974ea2117","Type":"ContainerStarted","Data":"5d173685b44700ea7449ca41100bc047eaf9e80eb3332f3f8823d1ec507e820c"}
Oct 30 20:55:07.957560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:07.957494    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89" podStartSLOduration=1.9574797529999999 podStartE2EDuration="1.957479753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:07.952669651 +0000 UTC m=+1074.444486195" watchObservedRunningTime="2026-10-30 20:55:07.957479753 +0000 UTC m=+1074.449296289"
Oct 30 20:55:08.943282 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:08.943247    4810 generic.go:356] "Generic (PLEG): container finished" podID="1124a4d8-de7b-45f0-8ffc-fd7974ea2117" containerID="69ced85f2764d73ed2142bc6a3d404909d4a83579f790febba277c9aa339d345" exitCode=0
Oct 30 20:55:08.943685 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:08.943342    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89" event={"ID":"1124a4d8-de7b-45f0-8ffc-fd7974ea2117","Type":"ContainerDied","Data":"69ced85f2764d73ed2142bc6a3d404909d4a83579f790febba277c9aa339d345"}
Oct 30 20:55:10.127611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.127584    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:10.218208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.218132    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-xks-ca\") pod \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") "
Oct 30 20:55:10.218477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.218247    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-openshift-ca\") pod \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") "
Oct 30 20:55:10.218477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.218335    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-kube-api-access-h9fj5\" (UniqueName: \"kubernetes.io/projected/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-kube-api-access-h9fj5\") pod \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\" (UID: \"1124a4d8-de7b-45f0-8ffc-fd7974ea2117\") "
Oct 30 20:55:10.218660 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.218634    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-openshift-ca" pod "1124a4d8-de7b-45f0-8ffc-fd7974ea2117" (UID: "1124a4d8-de7b-45f0-8ffc-fd7974ea2117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:10.218975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.218953    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:10.220911 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.220887    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-kube-api-access-h9fj5" pod "1124a4d8-de7b-45f0-8ffc-fd7974ea2117" (UID: "1124a4d8-de7b-45f0-8ffc-fd7974ea2117"). InnerVolumeSpecName "kube-api-access-h9fj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:10.221095 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.220934    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-xks-ca" pod "1124a4d8-de7b-45f0-8ffc-fd7974ea2117" (UID: "1124a4d8-de7b-45f0-8ffc-fd7974ea2117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:10.319782 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.319751    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:10.319782 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.319779    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9fj5\" (UniqueName: \"kubernetes.io/projected/1124a4d8-de7b-45f0-8ffc-fd7974ea2117-kube-api-access-h9fj5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:10.962892 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.962850    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"
Oct 30 20:55:10.962892 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.962888    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89" event={"ID":"1124a4d8-de7b-45f0-8ffc-fd7974ea2117","Type":"ContainerDied","Data":"5d173685b44700ea7449ca41100bc047eaf9e80eb3332f3f8823d1ec507e820c"}
Oct 30 20:55:10.963164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:10.962939    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d173685b44700ea7449ca41100bc047eaf9e80eb3332f3f8823d1ec507e820c"
Oct 30 20:55:14.268578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:14.268541    4810 scope.go:122] "RemoveContainer" containerID="8ff57fa8db13ca2949dc1d4d27ba072ca6826ae28d1ed3103ed615cd25668544"
Oct 30 20:55:14.280458 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:14.280437    4810 scope.go:122] "RemoveContainer" containerID="1bfb3bc7fd966d74b5dabfbe93f2ea75270dca94d9bfd3bbb215b10a966a7b8d"
Oct 30 20:55:14.292987 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:14.292966    4810 scope.go:122] "RemoveContainer" containerID="8760370e4f4cf0f8e0af85ef7af8651d86a8108d05ec36dd1d44c6401c8b0bb1"
Oct 30 20:55:25.154957 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:55:25.154914    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:55:26.170895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.170843    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:55:26.273119 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.273086    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"]
Oct 30 20:55:26.274117 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.274087    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1124a4d8-de7b-45f0-8ffc-fd7974ea2117" containerName="revoke-keys"
Oct 30 20:55:26.274117 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.274114    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1124a4d8-de7b-45f0-8ffc-fd7974ea2117" containerName="revoke-keys"
Oct 30 20:55:26.274340 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.274325    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1124a4d8-de7b-45f0-8ffc-fd7974ea2117" containerName="revoke-keys"
Oct 30 20:55:26.280427 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.280406    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:26.284949 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.284923    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7c14e2-dockercfg-8m29v\""
Oct 30 20:55:26.289961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.289939    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7c14e2\""
Oct 30 20:55:26.290360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.290340    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"]
Oct 30 20:55:26.312380 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.312350    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l\" (UID: \"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:26.312542 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.312493    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cpg5\" (UniqueName: \"kubernetes.io/projected/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-kube-api-access-9cpg5\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l\" (UID: \"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:26.321469 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.321443    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"]
Oct 30 20:55:26.347504 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.347041    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:26.350292 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.350262    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"]
Oct 30 20:55:26.416715 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.416672    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mmr\" (UniqueName: \"kubernetes.io/projected/db095638-5812-42ac-b3e2-2e3154bc5c3c-kube-api-access-69mmr\") pod \"payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69\" (UID: \"db095638-5812-42ac-b3e2-2e3154bc5c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:26.416922 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.416759    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l\" (UID: \"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:26.416922 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.416849    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db095638-5812-42ac-b3e2-2e3154bc5c3c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69\" (UID: \"db095638-5812-42ac-b3e2-2e3154bc5c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:26.417048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.417001    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cpg5\" (UniqueName: \"kubernetes.io/projected/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-kube-api-access-9cpg5\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l\" (UID: \"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:26.418343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.418313    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l\" (UID: \"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:26.427788 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.427726    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cpg5\" (UniqueName: \"kubernetes.io/projected/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-kube-api-access-9cpg5\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l\" (UID: \"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:26.519883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.519234    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mmr\" (UniqueName: \"kubernetes.io/projected/db095638-5812-42ac-b3e2-2e3154bc5c3c-kube-api-access-69mmr\") pod \"payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69\" (UID: \"db095638-5812-42ac-b3e2-2e3154bc5c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:26.519883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.519373    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db095638-5812-42ac-b3e2-2e3154bc5c3c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69\" (UID: \"db095638-5812-42ac-b3e2-2e3154bc5c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:26.537882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.520511    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db095638-5812-42ac-b3e2-2e3154bc5c3c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69\" (UID: \"db095638-5812-42ac-b3e2-2e3154bc5c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:26.547702 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.547663    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mmr\" (UniqueName: \"kubernetes.io/projected/db095638-5812-42ac-b3e2-2e3154bc5c3c-kube-api-access-69mmr\") pod \"payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69\" (UID: \"db095638-5812-42ac-b3e2-2e3154bc5c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:26.602410 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.602377    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:26.663688 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.663326    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:26.893969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:26.893938    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"]
Oct 30 20:55:26.894202 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:55:26.894175    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fb90f1_d3c4_418f_ac47_b55b892dbbdd.slice/crio-159bf3058b1ff405803a69761db762dd5b1dcd8e1d392fc74d54de52ee8836be WatchSource:0}: Error finding container 159bf3058b1ff405803a69761db762dd5b1dcd8e1d392fc74d54de52ee8836be: Status 404 returned error can't find the container with id 159bf3058b1ff405803a69761db762dd5b1dcd8e1d392fc74d54de52ee8836be
Oct 30 20:55:27.028629 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:27.028595    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"]
Oct 30 20:55:27.035236 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:55:27.035179    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb095638_5812_42ac_b3e2_2e3154bc5c3c.slice/crio-9571cc73394380bdcd1fbc98b75e794e328b499a632bb71e47a33be01bc45fa6 WatchSource:0}: Error finding container 9571cc73394380bdcd1fbc98b75e794e328b499a632bb71e47a33be01bc45fa6: Status 404 returned error can't find the container with id 9571cc73394380bdcd1fbc98b75e794e328b499a632bb71e47a33be01bc45fa6
Oct 30 20:55:27.113020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:27.112992    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l" event={"ID":"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd","Type":"ContainerStarted","Data":"56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d"}
Oct 30 20:55:27.113212 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:27.113042    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l" event={"ID":"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd","Type":"ContainerStarted","Data":"159bf3058b1ff405803a69761db762dd5b1dcd8e1d392fc74d54de52ee8836be"}
Oct 30 20:55:27.113829 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:27.113810    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:27.115525 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:27.115504    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69" event={"ID":"db095638-5812-42ac-b3e2-2e3154bc5c3c","Type":"ContainerStarted","Data":"9571cc73394380bdcd1fbc98b75e794e328b499a632bb71e47a33be01bc45fa6"}
Oct 30 20:55:27.137315 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:27.137275    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l" podStartSLOduration=1.137261179 podStartE2EDuration="1.137261179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:27.131894415 +0000 UTC m=+1093.623710960" watchObservedRunningTime="2026-10-30 20:55:27.137261179 +0000 UTC m=+1093.629077723"
Oct 30 20:55:28.125898 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.125843    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69" event={"ID":"db095638-5812-42ac-b3e2-2e3154bc5c3c","Type":"ContainerStarted","Data":"9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f"}
Oct 30 20:55:28.126371 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.126086    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:28.152543 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.152495    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69" podStartSLOduration=2.152480896 podStartE2EDuration="2.152480896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:28.147910967 +0000 UTC m=+1094.639727513" watchObservedRunningTime="2026-10-30 20:55:28.152480896 +0000 UTC m=+1094.644297469"
Oct 30 20:55:28.287539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.287491    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7615-4398ef"]
Oct 30 20:55:28.298494 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.296226    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-4398ef"
Oct 30 20:55:28.298494 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.297135    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7615-4398ef"]
Oct 30 20:55:28.300772 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.300732    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrzx8\""
Oct 30 20:55:28.371671 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.371635    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb4nw\" (UniqueName: \"kubernetes.io/projected/6c58510b-d54f-4bfc-ad7e-fa45834b5ad8-kube-api-access-hb4nw\") pod \"test-curl-7615-4398ef\" (UID: \"6c58510b-d54f-4bfc-ad7e-fa45834b5ad8\") " pod="opendatahub/test-curl-7615-4398ef"
Oct 30 20:55:28.472792 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.472759    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb4nw\" (UniqueName: \"kubernetes.io/projected/6c58510b-d54f-4bfc-ad7e-fa45834b5ad8-kube-api-access-hb4nw\") pod \"test-curl-7615-4398ef\" (UID: \"6c58510b-d54f-4bfc-ad7e-fa45834b5ad8\") " pod="opendatahub/test-curl-7615-4398ef"
Oct 30 20:55:28.481189 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.481163    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb4nw\" (UniqueName: \"kubernetes.io/projected/6c58510b-d54f-4bfc-ad7e-fa45834b5ad8-kube-api-access-hb4nw\") pod \"test-curl-7615-4398ef\" (UID: \"6c58510b-d54f-4bfc-ad7e-fa45834b5ad8\") " pod="opendatahub/test-curl-7615-4398ef"
Oct 30 20:55:28.635637 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.635604    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-4398ef"
Oct 30 20:55:28.928836 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:28.928808    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7615-4398ef"]
Oct 30 20:55:28.928971 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:55:28.928846    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c58510b_d54f_4bfc_ad7e_fa45834b5ad8.slice/crio-48070469ef9736959b5f59af65908a03a631eb1249722dcf9f00ebaed1909682 WatchSource:0}: Error finding container 48070469ef9736959b5f59af65908a03a631eb1249722dcf9f00ebaed1909682: Status 404 returned error can't find the container with id 48070469ef9736959b5f59af65908a03a631eb1249722dcf9f00ebaed1909682
Oct 30 20:55:29.135359 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:29.135328    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7615-4398ef" event={"ID":"6c58510b-d54f-4bfc-ad7e-fa45834b5ad8","Type":"ContainerStarted","Data":"47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7"}
Oct 30 20:55:29.135810 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:29.135375    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7615-4398ef" event={"ID":"6c58510b-d54f-4bfc-ad7e-fa45834b5ad8","Type":"ContainerStarted","Data":"48070469ef9736959b5f59af65908a03a631eb1249722dcf9f00ebaed1909682"}
Oct 30 20:55:29.150203 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:29.150116    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7615-4398ef" podStartSLOduration=1.150099427 podStartE2EDuration="1.150099427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:29.149758885 +0000 UTC m=+1095.641575436" watchObservedRunningTime="2026-10-30 20:55:29.150099427 +0000 UTC m=+1095.641915974"
Oct 30 20:55:29.817205 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:29.817155    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7615-4398ef"]
Oct 30 20:55:29.833729 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:29.833163    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7615-4398ef"]
Oct 30 20:55:30.182366 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.182331    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"]
Oct 30 20:55:30.183378 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.183344    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6c58510b-d54f-4bfc-ad7e-fa45834b5ad8" containerName="test-curl-7615-4398ef"
Oct 30 20:55:30.183378 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.183382    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c58510b-d54f-4bfc-ad7e-fa45834b5ad8" containerName="test-curl-7615-4398ef"
Oct 30 20:55:30.183539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.183527    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c58510b-d54f-4bfc-ad7e-fa45834b5ad8" containerName="test-curl-7615-4398ef"
Oct 30 20:55:30.187912 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.187886    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.192167 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.192117    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:55:30.193148 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.193121    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:55:30.193293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.193148    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"]
Oct 30 20:55:30.194744 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.194720    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.194873 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.194778    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n666z\" (UniqueName: \"kubernetes.io/projected/a2103470-a061-4bdf-9968-4ade51b5a721-kube-api-access-n666z\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.194948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.194877    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.296701 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.296662    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n666z\" (UniqueName: \"kubernetes.io/projected/a2103470-a061-4bdf-9968-4ade51b5a721-kube-api-access-n666z\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.296894 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.296749    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.297002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.296986    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.297574 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.297549    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.299407 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.299373    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.304644 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.304622    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n666z\" (UniqueName: \"kubernetes.io/projected/a2103470-a061-4bdf-9968-4ade51b5a721-kube-api-access-n666z\") pod \"maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.507187 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.507102    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:30.806754 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:30.806722    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"]
Oct 30 20:55:31.150932 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:31.150845    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57" event={"ID":"a2103470-a061-4bdf-9968-4ade51b5a721","Type":"ContainerStarted","Data":"7e6f9c2fb0b7f81581e8c9639d24879c3866e5640828928720a38ce1e6d00bf0"}
Oct 30 20:55:31.151103 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:31.151039    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7615-4398ef" podUID="6c58510b-d54f-4bfc-ad7e-fa45834b5ad8" containerName="test-curl-7615-4398ef" containerID="cri-o://47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7" gracePeriod=2
Oct 30 20:55:31.153381 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:55:31.153358    4810 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7615-4398ef\" is forbidden: User \"system:node:ip-10-0-128-161.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-161.us-east-2.compute.internal' and this object" podUID="6c58510b-d54f-4bfc-ad7e-fa45834b5ad8" pod="opendatahub/test-curl-7615-4398ef"
Oct 30 20:55:32.162086 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:32.162049    4810 generic.go:356] "Generic (PLEG): container finished" podID="a2103470-a061-4bdf-9968-4ade51b5a721" containerID="57b6fc5b9b44b523073def07df197be1adfdd57df55b9c3cb0bb38d3a9c86df1" exitCode=0
Oct 30 20:55:32.162537 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:32.162129    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57" event={"ID":"a2103470-a061-4bdf-9968-4ade51b5a721","Type":"ContainerDied","Data":"57b6fc5b9b44b523073def07df197be1adfdd57df55b9c3cb0bb38d3a9c86df1"}
Oct 30 20:55:33.383704 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.383666    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:33.387280 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.387257    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-4398ef"
Oct 30 20:55:33.456040 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.455962    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2103470-a061-4bdf-9968-4ade51b5a721-kube-api-access-n666z\" (UniqueName: \"kubernetes.io/projected/a2103470-a061-4bdf-9968-4ade51b5a721-kube-api-access-n666z\") pod \"a2103470-a061-4bdf-9968-4ade51b5a721\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") "
Oct 30 20:55:33.456040 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.456031    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c58510b-d54f-4bfc-ad7e-fa45834b5ad8-kube-api-access-hb4nw\" (UniqueName: \"kubernetes.io/projected/6c58510b-d54f-4bfc-ad7e-fa45834b5ad8-kube-api-access-hb4nw\") pod \"6c58510b-d54f-4bfc-ad7e-fa45834b5ad8\" (UID: \"6c58510b-d54f-4bfc-ad7e-fa45834b5ad8\") "
Oct 30 20:55:33.456235 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.456133    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-openshift-ca\") pod \"a2103470-a061-4bdf-9968-4ade51b5a721\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") "
Oct 30 20:55:33.456235 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.456180    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-xks-ca\") pod \"a2103470-a061-4bdf-9968-4ade51b5a721\" (UID: \"a2103470-a061-4bdf-9968-4ade51b5a721\") "
Oct 30 20:55:33.456527 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.456500    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-openshift-ca" pod "a2103470-a061-4bdf-9968-4ade51b5a721" (UID: "a2103470-a061-4bdf-9968-4ade51b5a721"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:33.456957 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.456932    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:33.458318 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.458286    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2103470-a061-4bdf-9968-4ade51b5a721-kube-api-access-n666z" pod "a2103470-a061-4bdf-9968-4ade51b5a721" (UID: "a2103470-a061-4bdf-9968-4ade51b5a721"). InnerVolumeSpecName "kube-api-access-n666z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:33.458393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.458325    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-xks-ca" pod "a2103470-a061-4bdf-9968-4ade51b5a721" (UID: "a2103470-a061-4bdf-9968-4ade51b5a721"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:33.458393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.458352    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c58510b-d54f-4bfc-ad7e-fa45834b5ad8-kube-api-access-hb4nw" pod "6c58510b-d54f-4bfc-ad7e-fa45834b5ad8" (UID: "6c58510b-d54f-4bfc-ad7e-fa45834b5ad8"). InnerVolumeSpecName "kube-api-access-hb4nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:33.558556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.558526    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n666z\" (UniqueName: \"kubernetes.io/projected/a2103470-a061-4bdf-9968-4ade51b5a721-kube-api-access-n666z\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:33.558556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.558550    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb4nw\" (UniqueName: \"kubernetes.io/projected/6c58510b-d54f-4bfc-ad7e-fa45834b5ad8-kube-api-access-hb4nw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:33.558556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.558560    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2103470-a061-4bdf-9968-4ade51b5a721-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:33.662912 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:33.662882    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c58510b-d54f-4bfc-ad7e-fa45834b5ad8" path="/var/lib/kubelet/pods/6c58510b-d54f-4bfc-ad7e-fa45834b5ad8/volumes"
Oct 30 20:55:34.187704 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:34.187656    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57" event={"ID":"a2103470-a061-4bdf-9968-4ade51b5a721","Type":"ContainerDied","Data":"7e6f9c2fb0b7f81581e8c9639d24879c3866e5640828928720a38ce1e6d00bf0"}
Oct 30 20:55:34.187921 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:34.187772    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e6f9c2fb0b7f81581e8c9639d24879c3866e5640828928720a38ce1e6d00bf0"
Oct 30 20:55:34.187921 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:34.187905    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"
Oct 30 20:55:34.192817 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:34.192787    4810 generic.go:356] "Generic (PLEG): container finished" podID="6c58510b-d54f-4bfc-ad7e-fa45834b5ad8" containerID="47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7" exitCode=137
Oct 30 20:55:34.193057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:34.193043    4810 scope.go:122] "RemoveContainer" containerID="47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7"
Oct 30 20:55:34.193568 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:34.193550    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-4398ef"
Oct 30 20:55:34.205242 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:34.205223    4810 scope.go:122] "RemoveContainer" containerID="47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7"
Oct 30 20:55:34.205531 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:55:34.205510    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7\": container with ID starting with 47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7 not found: ID does not exist" containerID="47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7"
Oct 30 20:55:34.205654 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:34.205538    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7"} err="failed to get container status \"47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7\": rpc error: code = NotFound desc = could not find container \"47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7\": container with ID starting with 47225720dc09bddb436dab09cd302450081584a992686d330afd3d0402d28df7 not found: ID does not exist"
Oct 30 20:55:37.336574 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.336535    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"]
Oct 30 20:55:37.337187 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.337171    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a2103470-a061-4bdf-9968-4ade51b5a721" containerName="revoke-keys"
Oct 30 20:55:37.337227 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.337188    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2103470-a061-4bdf-9968-4ade51b5a721" containerName="revoke-keys"
Oct 30 20:55:37.337304 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.337295    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2103470-a061-4bdf-9968-4ade51b5a721" containerName="revoke-keys"
Oct 30 20:55:37.342131 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.342113    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:55:37.349036 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.348968    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eebcb4\""
Oct 30 20:55:37.351222 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.351197    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"]
Oct 30 20:55:37.413545 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.413506    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l\" (UID: \"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:55:37.413787 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.413767    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdrb\" (UniqueName: \"kubernetes.io/projected/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-kube-api-access-6pdrb\") pod \"payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l\" (UID: \"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:55:37.417623 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.417596    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"]
Oct 30 20:55:37.422021 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.422002    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:55:37.426265 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.426244    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eebcb4-dockercfg-kscnc\""
Oct 30 20:55:37.430436 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.430414    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"]
Oct 30 20:55:37.524883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.524457    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l\" (UID: \"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:55:37.524883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.524525    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvpt\" (UniqueName: \"kubernetes.io/projected/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-kube-api-access-kkvpt\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj\" (UID: \"1e3e0257-7094-49a5-b25e-fdc4f8a9c847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:55:37.524883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.524671    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdrb\" (UniqueName: \"kubernetes.io/projected/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-kube-api-access-6pdrb\") pod \"payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l\" (UID: \"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:55:37.524883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.524700    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj\" (UID: \"1e3e0257-7094-49a5-b25e-fdc4f8a9c847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:55:37.530616 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.525627    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l\" (UID: \"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:55:37.589379 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.575550    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdrb\" (UniqueName: \"kubernetes.io/projected/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-kube-api-access-6pdrb\") pod \"payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l\" (UID: \"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:55:37.625817 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.625785    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvpt\" (UniqueName: \"kubernetes.io/projected/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-kube-api-access-kkvpt\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj\" (UID: \"1e3e0257-7094-49a5-b25e-fdc4f8a9c847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:55:37.626015 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.625958    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj\" (UID: \"1e3e0257-7094-49a5-b25e-fdc4f8a9c847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:55:37.626684 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.626663    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj\" (UID: \"1e3e0257-7094-49a5-b25e-fdc4f8a9c847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:55:37.635077 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.635054    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvpt\" (UniqueName: \"kubernetes.io/projected/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-kube-api-access-kkvpt\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj\" (UID: \"1e3e0257-7094-49a5-b25e-fdc4f8a9c847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:55:37.667293 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.667265    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:55:37.738967 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.738936    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:55:37.972238 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:55:37.972205    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fd16f2_d3cc_4b20_9ed6_cf2937d8e261.slice/crio-8ff45d0a5718c47e2cabbc563eed9a575bcdb5cef44fac5aa594cd3d9955bace WatchSource:0}: Error finding container 8ff45d0a5718c47e2cabbc563eed9a575bcdb5cef44fac5aa594cd3d9955bace: Status 404 returned error can't find the container with id 8ff45d0a5718c47e2cabbc563eed9a575bcdb5cef44fac5aa594cd3d9955bace
Oct 30 20:55:37.972468 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:37.972446    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"]
Oct 30 20:55:38.050912 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.050877    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"]
Oct 30 20:55:38.051313 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:55:38.051289    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3e0257_7094_49a5_b25e_fdc4f8a9c847.slice/crio-d6aff91f1035ac2172b72e8862048aa3cce86b718ac0b8eda8b1d6b49ce19a4f WatchSource:0}: Error finding container d6aff91f1035ac2172b72e8862048aa3cce86b718ac0b8eda8b1d6b49ce19a4f: Status 404 returned error can't find the container with id d6aff91f1035ac2172b72e8862048aa3cce86b718ac0b8eda8b1d6b49ce19a4f
Oct 30 20:55:38.264263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.264188    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj" event={"ID":"1e3e0257-7094-49a5-b25e-fdc4f8a9c847","Type":"ContainerStarted","Data":"d6aff91f1035ac2172b72e8862048aa3cce86b718ac0b8eda8b1d6b49ce19a4f"}
Oct 30 20:55:38.265682 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.265653    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l" event={"ID":"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261","Type":"ContainerStarted","Data":"8ff45d0a5718c47e2cabbc563eed9a575bcdb5cef44fac5aa594cd3d9955bace"}
Oct 30 20:55:38.604534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.604446    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"]
Oct 30 20:55:38.610007 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.609979    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:55:38.619370 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.619343    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5e71c9\""
Oct 30 20:55:38.623740 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.623690    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"]
Oct 30 20:55:38.653091 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.653063    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970ab33a-3c05-4f47-bde0-b2717b386da9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz\" (UID: \"970ab33a-3c05-4f47-bde0-b2717b386da9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:55:38.653326 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.653146    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2ns\" (UniqueName: \"kubernetes.io/projected/970ab33a-3c05-4f47-bde0-b2717b386da9-kube-api-access-tn2ns\") pod \"payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz\" (UID: \"970ab33a-3c05-4f47-bde0-b2717b386da9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:55:38.684971 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.684939    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"]
Oct 30 20:55:38.690308 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.690290    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:55:38.694467 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.694443    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5e71c9-dockercfg-7f6sd\""
Oct 30 20:55:38.698928 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.698829    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"]
Oct 30 20:55:38.816880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.758697    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970ab33a-3c05-4f47-bde0-b2717b386da9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz\" (UID: \"970ab33a-3c05-4f47-bde0-b2717b386da9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:55:38.816880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.758802    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2ns\" (UniqueName: \"kubernetes.io/projected/970ab33a-3c05-4f47-bde0-b2717b386da9-kube-api-access-tn2ns\") pod \"payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz\" (UID: \"970ab33a-3c05-4f47-bde0-b2717b386da9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:55:38.845882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.834620    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970ab33a-3c05-4f47-bde0-b2717b386da9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz\" (UID: \"970ab33a-3c05-4f47-bde0-b2717b386da9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:55:38.903881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.903745    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4h8h\" (UniqueName: \"kubernetes.io/projected/ccf11c95-fc2e-4d94-8cd8-402e743d8113-kube-api-access-j4h8h\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw\" (UID: \"ccf11c95-fc2e-4d94-8cd8-402e743d8113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:55:38.903881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.903877    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf11c95-fc2e-4d94-8cd8-402e743d8113-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw\" (UID: \"ccf11c95-fc2e-4d94-8cd8-402e743d8113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:55:38.935943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.932538    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2ns\" (UniqueName: \"kubernetes.io/projected/970ab33a-3c05-4f47-bde0-b2717b386da9-kube-api-access-tn2ns\") pod \"payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz\" (UID: \"970ab33a-3c05-4f47-bde0-b2717b386da9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:55:38.950348 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:38.950314    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:55:39.005355 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:39.005317    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4h8h\" (UniqueName: \"kubernetes.io/projected/ccf11c95-fc2e-4d94-8cd8-402e743d8113-kube-api-access-j4h8h\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw\" (UID: \"ccf11c95-fc2e-4d94-8cd8-402e743d8113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:55:39.005494 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:39.005420    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf11c95-fc2e-4d94-8cd8-402e743d8113-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw\" (UID: \"ccf11c95-fc2e-4d94-8cd8-402e743d8113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:55:39.006599 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:39.006578    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf11c95-fc2e-4d94-8cd8-402e743d8113-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw\" (UID: \"ccf11c95-fc2e-4d94-8cd8-402e743d8113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:55:39.015943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:39.015881    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4h8h\" (UniqueName: \"kubernetes.io/projected/ccf11c95-fc2e-4d94-8cd8-402e743d8113-kube-api-access-j4h8h\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw\" (UID: \"ccf11c95-fc2e-4d94-8cd8-402e743d8113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:55:39.140977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:39.140930    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:39.141165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:39.141012    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:39.282466 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:39.282436    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"]
Oct 30 20:55:39.328911 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:39.319185    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:55:39.671920 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:39.665544    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"]
Oct 30 20:55:39.671920 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:55:39.668054    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf11c95_fc2e_4d94_8cd8_402e743d8113.slice/crio-4ea3a757c600642eb9f74c2978023f8a3711d903048201321c04bf60ca45218b WatchSource:0}: Error finding container 4ea3a757c600642eb9f74c2978023f8a3711d903048201321c04bf60ca45218b: Status 404 returned error can't find the container with id 4ea3a757c600642eb9f74c2978023f8a3711d903048201321c04bf60ca45218b
Oct 30 20:55:40.297546 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:40.297457    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz" event={"ID":"970ab33a-3c05-4f47-bde0-b2717b386da9","Type":"ContainerStarted","Data":"706efba37ccd30439382c3a1f9c12f2bf2c98f36bdbc238f8ffb804f4bf4713b"}
Oct 30 20:55:40.304951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:40.304914    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw" event={"ID":"ccf11c95-fc2e-4d94-8cd8-402e743d8113","Type":"ContainerStarted","Data":"4ea3a757c600642eb9f74c2978023f8a3711d903048201321c04bf60ca45218b"}
Oct 30 20:55:41.039070 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:41.039023    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"]
Oct 30 20:55:41.046318 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:41.046289    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9b22a6e-4zr72"]
Oct 30 20:55:41.668571 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:41.668534    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86010140-5aad-4945-a9e7-4c46e180574f" path="/var/lib/kubelet/pods/86010140-5aad-4945-a9e7-4c46e180574f/volumes"
Oct 30 20:55:42.066906 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:42.066798    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"]
Oct 30 20:55:42.090309 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:42.082645    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a52bd2a1-sdb57"]
Oct 30 20:55:43.384626 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.384581    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz" event={"ID":"970ab33a-3c05-4f47-bde0-b2717b386da9","Type":"ContainerStarted","Data":"43d5927a3cfa6c93735e81e757984e5937d5eab3b2e321e0d44b1201d7a9ffac"}
Oct 30 20:55:43.385742 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.385720    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:55:43.399049 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.391186    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj" event={"ID":"1e3e0257-7094-49a5-b25e-fdc4f8a9c847","Type":"ContainerStarted","Data":"ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b"}
Oct 30 20:55:43.399049 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.393370    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:55:43.399049 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.395707    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw" event={"ID":"ccf11c95-fc2e-4d94-8cd8-402e743d8113","Type":"ContainerStarted","Data":"051cffc0fc99e0b4c912cb8cf3b9fddf73bca8e0719bf1432439b29c82d3acce"}
Oct 30 20:55:43.399049 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.396530    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:55:43.429877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.422947    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l" event={"ID":"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261","Type":"ContainerStarted","Data":"c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc"}
Oct 30 20:55:43.429877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.426339    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz" podStartSLOduration=2.196038596 podStartE2EDuration="5.426320794s" totalImagesPullingTime="3.230282198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:43.406636047 +0000 UTC m=+1109.898452594" watchObservedRunningTime="2026-10-30 20:55:43.426320794 +0000 UTC m=+1109.918137339"
Oct 30 20:55:43.439742 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.439698    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw" podStartSLOduration=2.52689761 podStartE2EDuration="5.439678297s" totalImagesPullingTime="2.912780687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:43.435187336 +0000 UTC m=+1109.927003891" watchObservedRunningTime="2026-10-30 20:55:43.439678297 +0000 UTC m=+1109.931494841"
Oct 30 20:55:43.458537 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.458483    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj" podStartSLOduration=2.157596266 podStartE2EDuration="6.458465624s" totalImagesPullingTime="4.300869358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:43.453595445 +0000 UTC m=+1109.945411991" watchObservedRunningTime="2026-10-30 20:55:43.458465624 +0000 UTC m=+1109.950282167"
Oct 30 20:55:43.472253 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.472175    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l" podStartSLOduration=2.014471407 podStartE2EDuration="6.472157805s" totalImagesPullingTime="4.457686398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:43.470280653 +0000 UTC m=+1109.962097200" watchObservedRunningTime="2026-10-30 20:55:43.472157805 +0000 UTC m=+1109.963974353"
Oct 30 20:55:43.664802 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:43.664729    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2103470-a061-4bdf-9968-4ade51b5a721" path="/var/lib/kubelet/pods/a2103470-a061-4bdf-9968-4ade51b5a721/volumes"
Oct 30 20:55:44.452919 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:44.448746    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:55:47.958802 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:47.958770    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:55:50.161528 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:50.161493    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"]
Oct 30 20:55:50.177233 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:50.177199    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"]
Oct 30 20:55:54.026095 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:54.026056    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"]
Oct 30 20:55:54.029938 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:54.029910    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd14b38e-xkvtn"]
Oct 30 20:55:55.047989 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.023052    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"]
Oct 30 20:55:55.051387 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.051347    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:55:55.056961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.056921    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"]
Oct 30 20:55:55.060088 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.060066    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7c14e2-dockercfg-ngpdp\""
Oct 30 20:55:55.155376 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.155334    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txkdw\" (UniqueName: \"kubernetes.io/projected/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-kube-api-access-txkdw\") pod \"payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp\" (UID: \"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:55:55.155567 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.155520    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp\" (UID: \"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:55:55.162561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.162534    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69" podUID="db095638-5812-42ac-b3e2-2e3154bc5c3c" containerName="payload-processing" containerID="cri-o://9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f" gracePeriod=25
Oct 30 20:55:55.179875 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.179021    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l" podUID="c4fb90f1-d3c4-418f-ac47-b55b892dbbdd" containerName="payload-pre-processing" containerID="cri-o://56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d" gracePeriod=25
Oct 30 20:55:55.196478 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:55:55.196446    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:55:55.256338 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.256295    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp\" (UID: \"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:55:55.256471 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.256454    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txkdw\" (UniqueName: \"kubernetes.io/projected/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-kube-api-access-txkdw\") pod \"payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp\" (UID: \"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:55:55.257022 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.257002    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp\" (UID: \"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:55:55.265894 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.265872    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txkdw\" (UniqueName: \"kubernetes.io/projected/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-kube-api-access-txkdw\") pod \"payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp\" (UID: \"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:55:55.461274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.461244    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:55:55.461742 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.461721    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:55:55.465712 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.465303    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:55:55.465712 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.465698    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:55:55.466191 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.465979    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:55:55.535773 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.535747    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:55.539384 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.539357    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:55.574937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.565259    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-kube-api-access-9cpg5\" (UniqueName: \"kubernetes.io/projected/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-kube-api-access-9cpg5\") pod \"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd\" (UID: \"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd\") "
Oct 30 20:55:55.574937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.565421    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db095638-5812-42ac-b3e2-2e3154bc5c3c-kube-api-access-69mmr\" (UniqueName: \"kubernetes.io/projected/db095638-5812-42ac-b3e2-2e3154bc5c3c-kube-api-access-69mmr\") pod \"db095638-5812-42ac-b3e2-2e3154bc5c3c\" (UID: \"db095638-5812-42ac-b3e2-2e3154bc5c3c\") "
Oct 30 20:55:55.574937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.565491    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-plugins-config-volume\") pod \"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd\" (UID: \"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd\") "
Oct 30 20:55:55.574937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.565541    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db095638-5812-42ac-b3e2-2e3154bc5c3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db095638-5812-42ac-b3e2-2e3154bc5c3c-plugins-config-volume\") pod \"db095638-5812-42ac-b3e2-2e3154bc5c3c\" (UID: \"db095638-5812-42ac-b3e2-2e3154bc5c3c\") "
Oct 30 20:55:55.574937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.570791    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-kube-api-access-9cpg5" pod "c4fb90f1-d3c4-418f-ac47-b55b892dbbdd" (UID: "c4fb90f1-d3c4-418f-ac47-b55b892dbbdd"). InnerVolumeSpecName "kube-api-access-9cpg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:55.581996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.578042    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db095638-5812-42ac-b3e2-2e3154bc5c3c-plugins-config-volume" pod "db095638-5812-42ac-b3e2-2e3154bc5c3c" (UID: "db095638-5812-42ac-b3e2-2e3154bc5c3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:55.581996 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.581264    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db095638-5812-42ac-b3e2-2e3154bc5c3c-kube-api-access-69mmr" pod "db095638-5812-42ac-b3e2-2e3154bc5c3c" (UID: "db095638-5812-42ac-b3e2-2e3154bc5c3c"). InnerVolumeSpecName "kube-api-access-69mmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:55.583369 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.583339    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-plugins-config-volume" pod "c4fb90f1-d3c4-418f-ac47-b55b892dbbdd" (UID: "c4fb90f1-d3c4-418f-ac47-b55b892dbbdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:55.630324 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.628555    4810 generic.go:356] "Generic (PLEG): container finished" podID="c4fb90f1-d3c4-418f-ac47-b55b892dbbdd" containerID="56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d" exitCode=0
Oct 30 20:55:55.630324 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.628791    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"
Oct 30 20:55:55.630324 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.628788    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l" event={"ID":"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd","Type":"ContainerDied","Data":"56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d"}
Oct 30 20:55:55.630324 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.628958    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l" event={"ID":"c4fb90f1-d3c4-418f-ac47-b55b892dbbdd","Type":"ContainerDied","Data":"159bf3058b1ff405803a69761db762dd5b1dcd8e1d392fc74d54de52ee8836be"}
Oct 30 20:55:55.630324 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.629002    4810 scope.go:122] "RemoveContainer" containerID="56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d"
Oct 30 20:55:55.645879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.638519    4810 generic.go:356] "Generic (PLEG): container finished" podID="db095638-5812-42ac-b3e2-2e3154bc5c3c" containerID="9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f" exitCode=0
Oct 30 20:55:55.645879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.638592    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69" event={"ID":"db095638-5812-42ac-b3e2-2e3154bc5c3c","Type":"ContainerDied","Data":"9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f"}
Oct 30 20:55:55.645879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.638639    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69" event={"ID":"db095638-5812-42ac-b3e2-2e3154bc5c3c","Type":"ContainerDied","Data":"9571cc73394380bdcd1fbc98b75e794e328b499a632bb71e47a33be01bc45fa6"}
Oct 30 20:55:55.645879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.638715    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"
Oct 30 20:55:55.646771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.646752    4810 scope.go:122] "RemoveContainer" containerID="56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d"
Oct 30 20:55:55.647142 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:55:55.647119    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d\": container with ID starting with 56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d not found: ID does not exist" containerID="56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d"
Oct 30 20:55:55.647234 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.647153    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d"} err="failed to get container status \"56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d\": rpc error: code = NotFound desc = could not find container \"56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d\": container with ID starting with 56689b70a06b99584d8bc3b40866d2ba01191fe50a7b212d63189e780f6fec8d not found: ID does not exist"
Oct 30 20:55:55.647234 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.647177    4810 scope.go:122] "RemoveContainer" containerID="9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f"
Oct 30 20:55:55.670023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.668839    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db095638-5812-42ac-b3e2-2e3154bc5c3c-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:55.670023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.668978    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cpg5\" (UniqueName: \"kubernetes.io/projected/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-kube-api-access-9cpg5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:55.670023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.668998    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69mmr\" (UniqueName: \"kubernetes.io/projected/db095638-5812-42ac-b3e2-2e3154bc5c3c-kube-api-access-69mmr\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:55.670023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.669013    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:55:55.670023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.669062    4810 scope.go:122] "RemoveContainer" containerID="9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f"
Oct 30 20:55:55.670023 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:55:55.669354    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f\": container with ID starting with 9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f not found: ID does not exist" containerID="9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f"
Oct 30 20:55:55.670023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.669386    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f"} err="failed to get container status \"9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f\": rpc error: code = NotFound desc = could not find container \"9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f\": container with ID starting with 9ee7beea845839d766b4449408c0c42651f58712ff3026fea136f860f62dd50f not found: ID does not exist"
Oct 30 20:55:55.672928 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.671111    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb3ccae-dfa1-48c3-8a52-268a035034ea" path="/var/lib/kubelet/pods/4eb3ccae-dfa1-48c3-8a52-268a035034ea/volumes"
Oct 30 20:55:55.672928 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.671525    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"]
Oct 30 20:55:55.678620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.678600    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-b567c487-x7f9l"]
Oct 30 20:55:55.692208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.692176    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"]
Oct 30 20:55:55.698722 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.698704    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-7ff887648f-fdm69"]
Oct 30 20:55:55.841684 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:55.841653    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"]
Oct 30 20:55:56.680140 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:56.680022    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp" event={"ID":"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1","Type":"ContainerStarted","Data":"9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7"}
Oct 30 20:55:56.682782 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:56.682751    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp" event={"ID":"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1","Type":"ContainerStarted","Data":"bb81dce5ecedeb6a9075a515a7f8dc048bb87240e66d1233cd454d824ffe2fb8"}
Oct 30 20:55:56.684305 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:56.684270    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:55:56.722074 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:56.718468    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp" podStartSLOduration=2.718447524 podStartE2EDuration="2.718447524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:56.713078267 +0000 UTC m=+1123.204894813" watchObservedRunningTime="2026-10-30 20:55:56.718447524 +0000 UTC m=+1123.210264072"
Oct 30 20:55:57.663280 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:57.663244    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4fb90f1-d3c4-418f-ac47-b55b892dbbdd" path="/var/lib/kubelet/pods/c4fb90f1-d3c4-418f-ac47-b55b892dbbdd/volumes"
Oct 30 20:55:57.663791 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:55:57.663760    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db095638-5812-42ac-b3e2-2e3154bc5c3c" path="/var/lib/kubelet/pods/db095638-5812-42ac-b3e2-2e3154bc5c3c/volumes"
Oct 30 20:56:04.036040 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:04.035998    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"]
Oct 30 20:56:04.040524 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:04.040498    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9591a6da-tn5sp"]
Oct 30 20:56:05.662849 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:05.662808    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940b3254-9a5c-4465-866c-c41234eea321" path="/var/lib/kubelet/pods/940b3254-9a5c-4465-866c-c41234eea321/volumes"
Oct 30 20:56:08.719958 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:08.719925    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:56:11.028654 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:11.028615    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"]
Oct 30 20:56:11.032239 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:11.032216    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9a6002-trskn"]
Oct 30 20:56:11.495089 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:11.495049    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:56:11.663025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:11.662993    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01517e7c-e24a-48a0-bfd3-72a9c5c03135" path="/var/lib/kubelet/pods/01517e7c-e24a-48a0-bfd3-72a9c5c03135/volumes"
Oct 30 20:56:14.372328 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.372293    4810 scope.go:122] "RemoveContainer" containerID="be2382cf46855d86f4d3e9daa4a298c1d2188742fc0b687a288f3ce8c8952a29"
Oct 30 20:56:14.384208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.384182    4810 scope.go:122] "RemoveContainer" containerID="26f1428769fe2082ca66f31e286b681de722ac61d8c8ee2f7c037667e8fa447c"
Oct 30 20:56:14.395836 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.395814    4810 scope.go:122] "RemoveContainer" containerID="5bb01d8f8ee6e4a7ff8779514a3ceed648fb4b208495736a65c36649bad67381"
Oct 30 20:56:14.625564 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.625476    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-kl7gz-rqftw"]
Oct 30 20:56:14.626269 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.626253    4810 cpu_manager.go:414] "containerMap: removing container" podUID="db095638-5812-42ac-b3e2-2e3154bc5c3c" containerName="payload-processing"
Oct 30 20:56:14.626343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.626270    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db095638-5812-42ac-b3e2-2e3154bc5c3c" containerName="payload-processing"
Oct 30 20:56:14.626343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.626293    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c4fb90f1-d3c4-418f-ac47-b55b892dbbdd" containerName="payload-pre-processing"
Oct 30 20:56:14.626343 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.626299    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4fb90f1-d3c4-418f-ac47-b55b892dbbdd" containerName="payload-pre-processing"
Oct 30 20:56:14.626459 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.626400    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="db095638-5812-42ac-b3e2-2e3154bc5c3c" containerName="payload-processing"
Oct 30 20:56:14.626459 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.626410    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4fb90f1-d3c4-418f-ac47-b55b892dbbdd" containerName="payload-pre-processing"
Oct 30 20:56:14.632250 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.632226    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.640830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.640803    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kl7gz-rqftw"]
Oct 30 20:56:14.782587 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.782544    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-service-ca-bundle\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.782771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.782693    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmm6\" (UniqueName: \"kubernetes.io/projected/23526b46-033a-49da-bbaa-72977ecc7645-kube-api-access-bbmm6\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.782906 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.782850    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/23526b46-033a-49da-bbaa-72977ecc7645-archives-path\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.783033 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.783015    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-trusted-ca-bundle\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.884325 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.884296    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmm6\" (UniqueName: \"kubernetes.io/projected/23526b46-033a-49da-bbaa-72977ecc7645-kube-api-access-bbmm6\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.884463 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.884437    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/23526b46-033a-49da-bbaa-72977ecc7645-archives-path\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.884534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.884515    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-trusted-ca-bundle\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.884583 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.884547    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-service-ca-bundle\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.884848 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.884826    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/23526b46-033a-49da-bbaa-72977ecc7645-archives-path\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.885297 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.885272    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-trusted-ca-bundle\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.885366 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.885317    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-service-ca-bundle\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.893083 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.893062    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmm6\" (UniqueName: \"kubernetes.io/projected/23526b46-033a-49da-bbaa-72977ecc7645-kube-api-access-bbmm6\") pod \"periodic-gathering-kl7gz-rqftw\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") " pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:14.951094 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:14.951068    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:56:15.245496 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:15.245398    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kl7gz-rqftw"]
Oct 30 20:56:15.270938 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:56:15.263936    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23526b46_033a_49da_bbaa_72977ecc7645.slice/crio-af7755e6061e223d84e1c4b79bde7012fc2a1d1a9215cdc68c3450c71e490c4d WatchSource:0}: Error finding container af7755e6061e223d84e1c4b79bde7012fc2a1d1a9215cdc68c3450c71e490c4d: Status 404 returned error can't find the container with id af7755e6061e223d84e1c4b79bde7012fc2a1d1a9215cdc68c3450c71e490c4d
Oct 30 20:56:16.029488 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:16.029454    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kl7gz-rqftw" event={"ID":"23526b46-033a-49da-bbaa-72977ecc7645","Type":"ContainerStarted","Data":"35405428b7bbc6885621445041bfe56650a1cbc9058176c12d6ad8c17b1bdbf1"}
Oct 30 20:56:16.029968 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:16.029507    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kl7gz-rqftw" event={"ID":"23526b46-033a-49da-bbaa-72977ecc7645","Type":"ContainerStarted","Data":"af7755e6061e223d84e1c4b79bde7012fc2a1d1a9215cdc68c3450c71e490c4d"}
Oct 30 20:56:16.045757 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:16.045711    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-kl7gz-rqftw" podStartSLOduration=2.045696909 podStartE2EDuration="2.045696909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:16.044788738 +0000 UTC m=+1142.536605290" watchObservedRunningTime="2026-10-30 20:56:16.045696909 +0000 UTC m=+1142.537513453"
Oct 30 20:56:17.930406 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:17.930368    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"]
Oct 30 20:56:17.934908 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:17.934888    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:17.940834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:17.940797    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-351fd1-dockercfg-xlzbp\""
Oct 30 20:56:17.940834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:17.940824    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-351fd1\""
Oct 30 20:56:17.945743 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:17.945720    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"]
Oct 30 20:56:17.969169 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:17.969141    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"]
Oct 30 20:56:17.973434 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:17.973404    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:17.988508 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:17.986938    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"]
Oct 30 20:56:18.140577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.140533    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf\" (UID: \"cb1d80e7-b0b6-414d-990f-c42d43c48d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:18.140743 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.140635    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxdv\" (UniqueName: \"kubernetes.io/projected/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-kube-api-access-9zxdv\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf\" (UID: \"cb1d80e7-b0b6-414d-990f-c42d43c48d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:18.140803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.140787    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53924db-eff9-43c8-8aaa-0aabf2a908f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf\" (UID: \"f53924db-eff9-43c8-8aaa-0aabf2a908f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:18.140908 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.140892    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpnkk\" (UniqueName: \"kubernetes.io/projected/f53924db-eff9-43c8-8aaa-0aabf2a908f3-kube-api-access-qpnkk\") pod \"payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf\" (UID: \"f53924db-eff9-43c8-8aaa-0aabf2a908f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:18.242627 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.242547    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpnkk\" (UniqueName: \"kubernetes.io/projected/f53924db-eff9-43c8-8aaa-0aabf2a908f3-kube-api-access-qpnkk\") pod \"payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf\" (UID: \"f53924db-eff9-43c8-8aaa-0aabf2a908f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:18.242627 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.242615    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf\" (UID: \"cb1d80e7-b0b6-414d-990f-c42d43c48d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:18.242804 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.242695    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxdv\" (UniqueName: \"kubernetes.io/projected/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-kube-api-access-9zxdv\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf\" (UID: \"cb1d80e7-b0b6-414d-990f-c42d43c48d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:18.243124 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.243098    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53924db-eff9-43c8-8aaa-0aabf2a908f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf\" (UID: \"f53924db-eff9-43c8-8aaa-0aabf2a908f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:18.243431 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.243412    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf\" (UID: \"cb1d80e7-b0b6-414d-990f-c42d43c48d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:18.243803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.243785    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53924db-eff9-43c8-8aaa-0aabf2a908f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf\" (UID: \"f53924db-eff9-43c8-8aaa-0aabf2a908f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:18.251570 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.251540    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxdv\" (UniqueName: \"kubernetes.io/projected/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-kube-api-access-9zxdv\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf\" (UID: \"cb1d80e7-b0b6-414d-990f-c42d43c48d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:18.251790 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.251775    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpnkk\" (UniqueName: \"kubernetes.io/projected/f53924db-eff9-43c8-8aaa-0aabf2a908f3-kube-api-access-qpnkk\") pod \"payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf\" (UID: \"f53924db-eff9-43c8-8aaa-0aabf2a908f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:18.298034 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.298001    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:18.551598 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.551501    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:18.614229 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.614099    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"]
Oct 30 20:56:18.865825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:18.865795    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"]
Oct 30 20:56:18.866008 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:56:18.865840    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1d80e7_b0b6_414d_990f_c42d43c48d0f.slice/crio-5fc1f61395b4a858d8d529354d65ee842da42abcf29dfb00d8d0d70f5dc61e06 WatchSource:0}: Error finding container 5fc1f61395b4a858d8d529354d65ee842da42abcf29dfb00d8d0d70f5dc61e06: Status 404 returned error can't find the container with id 5fc1f61395b4a858d8d529354d65ee842da42abcf29dfb00d8d0d70f5dc61e06
Oct 30 20:56:19.148454 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:19.148381    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf" event={"ID":"cb1d80e7-b0b6-414d-990f-c42d43c48d0f","Type":"ContainerStarted","Data":"664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5"}
Oct 30 20:56:19.148955 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:19.148554    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:19.148955 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:19.148584    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf" event={"ID":"cb1d80e7-b0b6-414d-990f-c42d43c48d0f","Type":"ContainerStarted","Data":"5fc1f61395b4a858d8d529354d65ee842da42abcf29dfb00d8d0d70f5dc61e06"}
Oct 30 20:56:19.150125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:19.150099    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf" event={"ID":"f53924db-eff9-43c8-8aaa-0aabf2a908f3","Type":"ContainerStarted","Data":"3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df"}
Oct 30 20:56:19.150234 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:19.150148    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:19.150234 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:19.150165    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf" event={"ID":"f53924db-eff9-43c8-8aaa-0aabf2a908f3","Type":"ContainerStarted","Data":"9229c637b92efb0c7f6c5edb7b1c0dfe777cfbb2f2d9c183bb695400bf53aaf1"}
Oct 30 20:56:19.169563 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:19.169519    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf" podStartSLOduration=2.169505889 podStartE2EDuration="2.169505889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:19.169033299 +0000 UTC m=+1145.660850048" watchObservedRunningTime="2026-10-30 20:56:19.169505889 +0000 UTC m=+1145.661322433"
Oct 30 20:56:19.187082 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:19.187028    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf" podStartSLOduration=2.187014281 podStartE2EDuration="2.187014281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:19.184915796 +0000 UTC m=+1145.676732339" watchObservedRunningTime="2026-10-30 20:56:19.187014281 +0000 UTC m=+1145.678830833"
Oct 30 20:56:20.025320 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:20.025286    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"]
Oct 30 20:56:20.028810 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:20.028786    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18d81d8-vtdd8"]
Oct 30 20:56:21.663910 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:21.663852    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb298f5-a1f6-42d5-9a81-bfeb4530d05e" path="/var/lib/kubelet/pods/2cb298f5-a1f6-42d5-9a81-bfeb4530d05e/volumes"
Oct 30 20:56:24.051977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:24.040922    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"]
Oct 30 20:56:24.051977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:24.049989    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a7d130-b4tnp"]
Oct 30 20:56:25.208473 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:56:25.207512    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:56:25.699883 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:25.699248    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45204b7-d73e-4c52-b047-d3b107f99e25" path="/var/lib/kubelet/pods/b45204b7-d73e-4c52-b047-d3b107f99e25/volumes"
Oct 30 20:56:30.161771 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:30.161739    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:30.162214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:30.161824    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:32.401718 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.401678    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"]
Oct 30 20:56:32.406371 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.406347    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:56:32.420465 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.419812    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"]
Oct 30 20:56:32.420991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.420966    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5452c3\""
Oct 30 20:56:32.479606 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.479573    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"]
Oct 30 20:56:32.484803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.484774    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:56:32.495783 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.495753    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5452c3-dockercfg-x8pfl\""
Oct 30 20:56:32.501071 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.501043    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"]
Oct 30 20:56:32.551371 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.539058    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c4609a-1527-414e-b729-ffdf0c080407-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm\" (UID: \"41c4609a-1527-414e-b729-ffdf0c080407\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:56:32.551371 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.539119    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6785af5-f24c-43f6-a350-3f79f80e49c3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6\" (UID: \"c6785af5-f24c-43f6-a350-3f79f80e49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:56:32.551371 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.539155    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v52t7\" (UniqueName: \"kubernetes.io/projected/41c4609a-1527-414e-b729-ffdf0c080407-kube-api-access-v52t7\") pod \"payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm\" (UID: \"41c4609a-1527-414e-b729-ffdf0c080407\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:56:32.551371 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.539245    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-745f4\" (UniqueName: \"kubernetes.io/projected/c6785af5-f24c-43f6-a350-3f79f80e49c3-kube-api-access-745f4\") pod \"payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6\" (UID: \"c6785af5-f24c-43f6-a350-3f79f80e49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:56:32.640741 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.640695    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c4609a-1527-414e-b729-ffdf0c080407-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm\" (UID: \"41c4609a-1527-414e-b729-ffdf0c080407\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:56:32.640939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.640749    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6785af5-f24c-43f6-a350-3f79f80e49c3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6\" (UID: \"c6785af5-f24c-43f6-a350-3f79f80e49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:56:32.640939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.640780    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v52t7\" (UniqueName: \"kubernetes.io/projected/41c4609a-1527-414e-b729-ffdf0c080407-kube-api-access-v52t7\") pod \"payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm\" (UID: \"41c4609a-1527-414e-b729-ffdf0c080407\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:56:32.640939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.640879    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-745f4\" (UniqueName: \"kubernetes.io/projected/c6785af5-f24c-43f6-a350-3f79f80e49c3-kube-api-access-745f4\") pod \"payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6\" (UID: \"c6785af5-f24c-43f6-a350-3f79f80e49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:56:32.641640 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.641613    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6785af5-f24c-43f6-a350-3f79f80e49c3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6\" (UID: \"c6785af5-f24c-43f6-a350-3f79f80e49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:56:32.642505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.642478    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c4609a-1527-414e-b729-ffdf0c080407-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm\" (UID: \"41c4609a-1527-414e-b729-ffdf0c080407\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:56:32.650477 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.650448    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v52t7\" (UniqueName: \"kubernetes.io/projected/41c4609a-1527-414e-b729-ffdf0c080407-kube-api-access-v52t7\") pod \"payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm\" (UID: \"41c4609a-1527-414e-b729-ffdf0c080407\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:56:32.651199 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.651174    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-745f4\" (UniqueName: \"kubernetes.io/projected/c6785af5-f24c-43f6-a350-3f79f80e49c3-kube-api-access-745f4\") pod \"payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6\" (UID: \"c6785af5-f24c-43f6-a350-3f79f80e49c3\") " pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:56:32.731969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.728145    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:56:32.803175 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:32.803130    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:56:33.030295 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:33.030246    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"]
Oct 30 20:56:33.036799 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:33.036765    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e95bf2f-lp5ds"]
Oct 30 20:56:33.097655 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:33.097617    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"]
Oct 30 20:56:33.232969 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:33.209514    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"]
Oct 30 20:56:33.353025 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:33.352992    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6" event={"ID":"c6785af5-f24c-43f6-a350-3f79f80e49c3","Type":"ContainerStarted","Data":"9368bece9fe55c8947405fc2bc6cace9fe3374ed71e042564c96805dc857f52d"}
Oct 30 20:56:33.356724 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:33.356692    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:56:33.357359 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:33.357287    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm" event={"ID":"41c4609a-1527-414e-b729-ffdf0c080407","Type":"ContainerStarted","Data":"74093501d0ba9bee38c90a374e0855c677c90116d632b3ca3a68fdb10fed0ac2"}
Oct 30 20:56:33.357500 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:33.357396    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm" event={"ID":"41c4609a-1527-414e-b729-ffdf0c080407","Type":"ContainerStarted","Data":"a4e000e71968a26d24556d2601e6889f992c43bf3afba4d94c8205525b49e577"}
Oct 30 20:56:33.381004 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:33.380912    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm" podStartSLOduration=1.380892213 podStartE2EDuration="1.380892213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:33.375366567 +0000 UTC m=+1159.867183116" watchObservedRunningTime="2026-10-30 20:56:33.380892213 +0000 UTC m=+1159.872708757"
Oct 30 20:56:33.688676 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:33.688643    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48daa322-4980-4fd8-a2b7-acfe795df6e6" path="/var/lib/kubelet/pods/48daa322-4980-4fd8-a2b7-acfe795df6e6/volumes"
Oct 30 20:56:34.369426 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:34.369385    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6" event={"ID":"c6785af5-f24c-43f6-a350-3f79f80e49c3","Type":"ContainerStarted","Data":"88875e12238e72885e01ce0cc0b544791de4fbd3e11d7a9a9b6b4bb2e07afb75"}
Oct 30 20:56:34.395014 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:34.394964    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6" podStartSLOduration=2.394950334 podStartE2EDuration="2.394950334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:34.394317338 +0000 UTC m=+1160.886133884" watchObservedRunningTime="2026-10-30 20:56:34.394950334 +0000 UTC m=+1160.886766913"
Oct 30 20:56:35.379176 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:35.379144    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:56:40.526404 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.526373    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"]
Oct 30 20:56:40.531991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.531975    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:56:40.536433 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.536404    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c5bf9a-dockercfg-75nnr\""
Oct 30 20:56:40.536989 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.536967    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c5bf9a\""
Oct 30 20:56:40.540521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.540498    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"]
Oct 30 20:56:40.705880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.705009    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"]
Oct 30 20:56:40.791329 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.791238    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67n5b\" (UniqueName: \"kubernetes.io/projected/4572061f-d2e5-4d59-ad02-ef7cc08a016e-kube-api-access-67n5b\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj\" (UID: \"4572061f-d2e5-4d59-ad02-ef7cc08a016e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:56:40.791508 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.791326    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4572061f-d2e5-4d59-ad02-ef7cc08a016e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj\" (UID: \"4572061f-d2e5-4d59-ad02-ef7cc08a016e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:56:40.805352 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.803686    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"]
Oct 30 20:56:40.805352 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.803808    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:56:40.893331 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.893292    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67n5b\" (UniqueName: \"kubernetes.io/projected/4572061f-d2e5-4d59-ad02-ef7cc08a016e-kube-api-access-67n5b\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj\" (UID: \"4572061f-d2e5-4d59-ad02-ef7cc08a016e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:56:40.893531 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.893370    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4572061f-d2e5-4d59-ad02-ef7cc08a016e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj\" (UID: \"4572061f-d2e5-4d59-ad02-ef7cc08a016e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:56:40.893531 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.893501    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln\" (UID: \"9e84eb10-6a53-46ad-a1ec-5df89a4241e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:56:40.893662 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.893565    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpw7t\" (UniqueName: \"kubernetes.io/projected/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-kube-api-access-zpw7t\") pod \"payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln\" (UID: \"9e84eb10-6a53-46ad-a1ec-5df89a4241e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:56:40.894190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.894164    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4572061f-d2e5-4d59-ad02-ef7cc08a016e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj\" (UID: \"4572061f-d2e5-4d59-ad02-ef7cc08a016e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:56:40.903534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.903509    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67n5b\" (UniqueName: \"kubernetes.io/projected/4572061f-d2e5-4d59-ad02-ef7cc08a016e-kube-api-access-67n5b\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj\" (UID: \"4572061f-d2e5-4d59-ad02-ef7cc08a016e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:56:40.994644 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.994612    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpw7t\" (UniqueName: \"kubernetes.io/projected/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-kube-api-access-zpw7t\") pod \"payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln\" (UID: \"9e84eb10-6a53-46ad-a1ec-5df89a4241e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:56:40.994844 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.994828    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln\" (UID: \"9e84eb10-6a53-46ad-a1ec-5df89a4241e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:56:40.995734 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:40.995711    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln\" (UID: \"9e84eb10-6a53-46ad-a1ec-5df89a4241e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:56:41.003539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:41.003510    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpw7t\" (UniqueName: \"kubernetes.io/projected/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-kube-api-access-zpw7t\") pod \"payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln\" (UID: \"9e84eb10-6a53-46ad-a1ec-5df89a4241e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:56:41.122123 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:41.122047    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:56:41.149438 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:41.149398    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:56:41.407590 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:56:41.407546    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e84eb10_6a53_46ad_a1ec_5df89a4241e6.slice/crio-4d55050cce1998fcf795ab711529b08a4199e348ba656530b445b0a1a0a9d5b8 WatchSource:0}: Error finding container 4d55050cce1998fcf795ab711529b08a4199e348ba656530b445b0a1a0a9d5b8: Status 404 returned error can't find the container with id 4d55050cce1998fcf795ab711529b08a4199e348ba656530b445b0a1a0a9d5b8
Oct 30 20:56:41.408126 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:41.408095    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"]
Oct 30 20:56:41.433138 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:41.433107    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln" event={"ID":"9e84eb10-6a53-46ad-a1ec-5df89a4241e6","Type":"ContainerStarted","Data":"4d55050cce1998fcf795ab711529b08a4199e348ba656530b445b0a1a0a9d5b8"}
Oct 30 20:56:41.481780 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:56:41.481719    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4572061f_d2e5_4d59_ad02_ef7cc08a016e.slice/crio-f0c0af8b0da036fe320736ab90343f803b87963a2a461f87a6480b6346264f75 WatchSource:0}: Error finding container f0c0af8b0da036fe320736ab90343f803b87963a2a461f87a6480b6346264f75: Status 404 returned error can't find the container with id f0c0af8b0da036fe320736ab90343f803b87963a2a461f87a6480b6346264f75
Oct 30 20:56:41.485603 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:41.485577    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"]
Oct 30 20:56:42.451433 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:42.451341    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj" event={"ID":"4572061f-d2e5-4d59-ad02-ef7cc08a016e","Type":"ContainerStarted","Data":"e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1"}
Oct 30 20:56:42.451433 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:42.451398    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj" event={"ID":"4572061f-d2e5-4d59-ad02-ef7cc08a016e","Type":"ContainerStarted","Data":"f0c0af8b0da036fe320736ab90343f803b87963a2a461f87a6480b6346264f75"}
Oct 30 20:56:42.453137 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:42.452148    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:56:42.454611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:42.454579    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln" event={"ID":"9e84eb10-6a53-46ad-a1ec-5df89a4241e6","Type":"ContainerStarted","Data":"9e76b70f372047b572321441702d539b7e11d9ddf2c8785b9a6c2e4a9bd3ba3d"}
Oct 30 20:56:42.454746 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:42.454635    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:56:42.498884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:42.494711    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln" podStartSLOduration=2.494693609 podStartE2EDuration="2.494693609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:42.492239328 +0000 UTC m=+1168.984055874" watchObservedRunningTime="2026-10-30 20:56:42.494693609 +0000 UTC m=+1168.986510154"
Oct 30 20:56:42.498884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:42.496161    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj" podStartSLOduration=2.496145639 podStartE2EDuration="2.496145639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:42.472016426 +0000 UTC m=+1168.963832974" watchObservedRunningTime="2026-10-30 20:56:42.496145639 +0000 UTC m=+1168.987962187"
Oct 30 20:56:44.032066 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:44.032027    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"]
Oct 30 20:56:44.035831 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:44.035802    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d85271a-6dbkq"]
Oct 30 20:56:45.407882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:45.406656    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:56:45.713634 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:45.713554    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a92702f-9cea-4251-a2d2-b873bf2898d7" path="/var/lib/kubelet/pods/3a92702f-9cea-4251-a2d2-b873bf2898d7/volumes"
Oct 30 20:56:46.391886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:46.391848    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:56:48.002268 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:48.002228    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:56:52.193279 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:52.193245    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"]
Oct 30 20:56:52.206992 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:52.206964    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"]
Oct 30 20:56:53.465614 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:53.465585    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:56:53.466094 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:53.465691    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:56:54.026987 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:54.026943    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"]
Oct 30 20:56:54.030988 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:54.030958    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95edb6c8-hfqx5"]
Oct 30 20:56:55.157378 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:56:55.157276    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:56:55.663427 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.663399    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab448cbe-adc2-4113-a5ed-0dc76a012875" path="/var/lib/kubelet/pods/ab448cbe-adc2-4113-a5ed-0dc76a012875/volumes"
Oct 30 20:56:55.701665 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.701629    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"]
Oct 30 20:56:55.706193 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.706177    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:55.715648 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.715619    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"]
Oct 30 20:56:55.719414 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.719390    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:56:55.719526 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.719480    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:56:55.837948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.837914    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:55.838127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.837969    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:55.838127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.838067    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6v5\" (UniqueName: \"kubernetes.io/projected/40f46cb5-ba9f-47f4-b431-01da37f74f42-kube-api-access-nv6v5\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:55.939329 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.939250    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:55.939329 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.939314    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6v5\" (UniqueName: \"kubernetes.io/projected/40f46cb5-ba9f-47f4-b431-01da37f74f42-kube-api-access-nv6v5\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:55.939505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.939467    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:55.940104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.940085    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:55.941651 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.941625    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:55.947751 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:55.947726    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6v5\" (UniqueName: \"kubernetes.io/projected/40f46cb5-ba9f-47f4-b431-01da37f74f42-kube-api-access-nv6v5\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:56.024991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:56.024963    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:56.314164 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:56:56.314132    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f46cb5_ba9f_47f4_b431_01da37f74f42.slice/crio-9eb16bbf19d4e0b621c4e8a6b6d9385fb8ea3a2ff2168186aab483e06eda6f8e WatchSource:0}: Error finding container 9eb16bbf19d4e0b621c4e8a6b6d9385fb8ea3a2ff2168186aab483e06eda6f8e: Status 404 returned error can't find the container with id 9eb16bbf19d4e0b621c4e8a6b6d9385fb8ea3a2ff2168186aab483e06eda6f8e
Oct 30 20:56:56.314530 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:56.314226    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"]
Oct 30 20:56:56.647670 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:56.647637    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n" event={"ID":"40f46cb5-ba9f-47f4-b431-01da37f74f42","Type":"ContainerStarted","Data":"9eb16bbf19d4e0b621c4e8a6b6d9385fb8ea3a2ff2168186aab483e06eda6f8e"}
Oct 30 20:56:57.193851 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.193818    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf" podUID="f53924db-eff9-43c8-8aaa-0aabf2a908f3" containerName="payload-processing" containerID="cri-o://3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df" gracePeriod=25
Oct 30 20:56:57.207566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.207519    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf" podUID="cb1d80e7-b0b6-414d-990f-c42d43c48d0f" containerName="payload-pre-processing" containerID="cri-o://664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5" gracePeriod=25
Oct 30 20:56:57.511428 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.511401    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:57.523274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.523246    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:57.666763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.666729    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-plugins-config-volume\") pod \"cb1d80e7-b0b6-414d-990f-c42d43c48d0f\" (UID: \"cb1d80e7-b0b6-414d-990f-c42d43c48d0f\") "
Oct 30 20:56:57.666982 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.666851    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-kube-api-access-9zxdv\" (UniqueName: \"kubernetes.io/projected/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-kube-api-access-9zxdv\") pod \"cb1d80e7-b0b6-414d-990f-c42d43c48d0f\" (UID: \"cb1d80e7-b0b6-414d-990f-c42d43c48d0f\") "
Oct 30 20:56:57.666982 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.666927    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53924db-eff9-43c8-8aaa-0aabf2a908f3-kube-api-access-qpnkk\" (UniqueName: \"kubernetes.io/projected/f53924db-eff9-43c8-8aaa-0aabf2a908f3-kube-api-access-qpnkk\") pod \"f53924db-eff9-43c8-8aaa-0aabf2a908f3\" (UID: \"f53924db-eff9-43c8-8aaa-0aabf2a908f3\") "
Oct 30 20:56:57.667104 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.667064    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53924db-eff9-43c8-8aaa-0aabf2a908f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53924db-eff9-43c8-8aaa-0aabf2a908f3-plugins-config-volume\") pod \"f53924db-eff9-43c8-8aaa-0aabf2a908f3\" (UID: \"f53924db-eff9-43c8-8aaa-0aabf2a908f3\") "
Oct 30 20:56:57.667220 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.667190    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-plugins-config-volume" pod "cb1d80e7-b0b6-414d-990f-c42d43c48d0f" (UID: "cb1d80e7-b0b6-414d-990f-c42d43c48d0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:57.667967 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.667562    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53924db-eff9-43c8-8aaa-0aabf2a908f3-plugins-config-volume" pod "f53924db-eff9-43c8-8aaa-0aabf2a908f3" (UID: "f53924db-eff9-43c8-8aaa-0aabf2a908f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:57.668120 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.667635    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:56:57.669332 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.669308    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53924db-eff9-43c8-8aaa-0aabf2a908f3-kube-api-access-qpnkk" pod "f53924db-eff9-43c8-8aaa-0aabf2a908f3" (UID: "f53924db-eff9-43c8-8aaa-0aabf2a908f3"). InnerVolumeSpecName "kube-api-access-qpnkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:57.673005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.672945    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-kube-api-access-9zxdv" pod "cb1d80e7-b0b6-414d-990f-c42d43c48d0f" (UID: "cb1d80e7-b0b6-414d-990f-c42d43c48d0f"). InnerVolumeSpecName "kube-api-access-9zxdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:57.674294 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.674267    4810 generic.go:356] "Generic (PLEG): container finished" podID="f53924db-eff9-43c8-8aaa-0aabf2a908f3" containerID="3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df" exitCode=0
Oct 30 20:56:57.674408 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.674389    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"
Oct 30 20:56:57.675432 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.675406    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf" event={"ID":"f53924db-eff9-43c8-8aaa-0aabf2a908f3","Type":"ContainerDied","Data":"3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df"}
Oct 30 20:56:57.675578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.675459    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf" event={"ID":"f53924db-eff9-43c8-8aaa-0aabf2a908f3","Type":"ContainerDied","Data":"9229c637b92efb0c7f6c5edb7b1c0dfe777cfbb2f2d9c183bb695400bf53aaf1"}
Oct 30 20:56:57.675578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.675489    4810 scope.go:122] "RemoveContainer" containerID="3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df"
Oct 30 20:56:57.676850 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.676826    4810 generic.go:356] "Generic (PLEG): container finished" podID="cb1d80e7-b0b6-414d-990f-c42d43c48d0f" containerID="664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5" exitCode=0
Oct 30 20:56:57.677951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.676962    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf" event={"ID":"cb1d80e7-b0b6-414d-990f-c42d43c48d0f","Type":"ContainerDied","Data":"664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5"}
Oct 30 20:56:57.677951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.677004    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf" event={"ID":"cb1d80e7-b0b6-414d-990f-c42d43c48d0f","Type":"ContainerDied","Data":"5fc1f61395b4a858d8d529354d65ee842da42abcf29dfb00d8d0d70f5dc61e06"}
Oct 30 20:56:57.677951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.677104    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"
Oct 30 20:56:57.680649 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.680627    4810 generic.go:356] "Generic (PLEG): container finished" podID="40f46cb5-ba9f-47f4-b431-01da37f74f42" containerID="ff2171dc0953b2c12fe9fe91ba98cd8d8a34d2b8a21096eb6953b586162feb84" exitCode=0
Oct 30 20:56:57.680744 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.680699    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n" event={"ID":"40f46cb5-ba9f-47f4-b431-01da37f74f42","Type":"ContainerDied","Data":"ff2171dc0953b2c12fe9fe91ba98cd8d8a34d2b8a21096eb6953b586162feb84"}
Oct 30 20:56:57.690138 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.690027    4810 scope.go:122] "RemoveContainer" containerID="3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df"
Oct 30 20:56:57.690429 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:56:57.690407    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df\": container with ID starting with 3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df not found: ID does not exist" containerID="3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df"
Oct 30 20:56:57.690490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.690434    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df"} err="failed to get container status \"3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df\": rpc error: code = NotFound desc = could not find container \"3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df\": container with ID starting with 3767162c4e893b79c2b0b574b3ee9266f5ae066ba789b0331e914a440ce133df not found: ID does not exist"
Oct 30 20:56:57.690490 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.690450    4810 scope.go:122] "RemoveContainer" containerID="664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5"
Oct 30 20:56:57.703570 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.703551    4810 scope.go:122] "RemoveContainer" containerID="664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5"
Oct 30 20:56:57.703928 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:56:57.703846    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5\": container with ID starting with 664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5 not found: ID does not exist" containerID="664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5"
Oct 30 20:56:57.704021 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.703982    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5"} err="failed to get container status \"664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5\": rpc error: code = NotFound desc = could not find container \"664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5\": container with ID starting with 664d21d69ea822ad8d691a4ec73e9480121799550054b3b3d85c7a4b223c4bf5 not found: ID does not exist"
Oct 30 20:56:57.723164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.723112    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"]
Oct 30 20:56:57.726838 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.726791    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-69c9d6c8ff-rqscf"]
Oct 30 20:56:57.737109 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.737089    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"]
Oct 30 20:56:57.743604 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.743583    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-584c5df55-df5nf"]
Oct 30 20:56:57.769448 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.769420    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zxdv\" (UniqueName: \"kubernetes.io/projected/cb1d80e7-b0b6-414d-990f-c42d43c48d0f-kube-api-access-9zxdv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:56:57.769448 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.769448    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpnkk\" (UniqueName: \"kubernetes.io/projected/f53924db-eff9-43c8-8aaa-0aabf2a908f3-kube-api-access-qpnkk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:56:57.769620 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:57.769462    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f53924db-eff9-43c8-8aaa-0aabf2a908f3-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:56:58.863672 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:58.863648    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:58.985411 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:58.985330    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f46cb5-ba9f-47f4-b431-01da37f74f42-kube-api-access-nv6v5\" (UniqueName: \"kubernetes.io/projected/40f46cb5-ba9f-47f4-b431-01da37f74f42-kube-api-access-nv6v5\") pod \"40f46cb5-ba9f-47f4-b431-01da37f74f42\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") "
Oct 30 20:56:58.985551 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:58.985502    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-openshift-ca\") pod \"40f46cb5-ba9f-47f4-b431-01da37f74f42\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") "
Oct 30 20:56:58.985640 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:58.985620    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-xks-ca\") pod \"40f46cb5-ba9f-47f4-b431-01da37f74f42\" (UID: \"40f46cb5-ba9f-47f4-b431-01da37f74f42\") "
Oct 30 20:56:58.985947 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:58.985906    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-openshift-ca" pod "40f46cb5-ba9f-47f4-b431-01da37f74f42" (UID: "40f46cb5-ba9f-47f4-b431-01da37f74f42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:58.986183 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:58.986159    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:56:58.987686 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:58.987653    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-xks-ca" pod "40f46cb5-ba9f-47f4-b431-01da37f74f42" (UID: "40f46cb5-ba9f-47f4-b431-01da37f74f42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:56:58.987906 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:58.987848    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f46cb5-ba9f-47f4-b431-01da37f74f42-kube-api-access-nv6v5" pod "40f46cb5-ba9f-47f4-b431-01da37f74f42" (UID: "40f46cb5-ba9f-47f4-b431-01da37f74f42"). InnerVolumeSpecName "kube-api-access-nv6v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:59.087007 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.086981    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f46cb5-ba9f-47f4-b431-01da37f74f42-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:56:59.087007 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.087006    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv6v5\" (UniqueName: \"kubernetes.io/projected/40f46cb5-ba9f-47f4-b431-01da37f74f42-kube-api-access-nv6v5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:56:59.663426 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.663396    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1d80e7-b0b6-414d-990f-c42d43c48d0f" path="/var/lib/kubelet/pods/cb1d80e7-b0b6-414d-990f-c42d43c48d0f/volumes"
Oct 30 20:56:59.663794 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.663779    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53924db-eff9-43c8-8aaa-0aabf2a908f3" path="/var/lib/kubelet/pods/f53924db-eff9-43c8-8aaa-0aabf2a908f3/volumes"
Oct 30 20:56:59.703568 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.703541    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"
Oct 30 20:56:59.703568 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.703559    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n" event={"ID":"40f46cb5-ba9f-47f4-b431-01da37f74f42","Type":"ContainerDied","Data":"9eb16bbf19d4e0b621c4e8a6b6d9385fb8ea3a2ff2168186aab483e06eda6f8e"}
Oct 30 20:56:59.703807 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.703607    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eb16bbf19d4e0b621c4e8a6b6d9385fb8ea3a2ff2168186aab483e06eda6f8e"
Oct 30 20:56:59.882189 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.882160    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"]
Oct 30 20:56:59.882821 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.882807    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f53924db-eff9-43c8-8aaa-0aabf2a908f3" containerName="payload-processing"
Oct 30 20:56:59.882879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.882822    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53924db-eff9-43c8-8aaa-0aabf2a908f3" containerName="payload-processing"
Oct 30 20:56:59.882879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.882843    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cb1d80e7-b0b6-414d-990f-c42d43c48d0f" containerName="payload-pre-processing"
Oct 30 20:56:59.882879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.882848    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1d80e7-b0b6-414d-990f-c42d43c48d0f" containerName="payload-pre-processing"
Oct 30 20:56:59.882879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.882868    4810 cpu_manager.go:414] "containerMap: removing container" podUID="40f46cb5-ba9f-47f4-b431-01da37f74f42" containerName="revoke-keys"
Oct 30 20:56:59.882879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.882874    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f46cb5-ba9f-47f4-b431-01da37f74f42" containerName="revoke-keys"
Oct 30 20:56:59.883033 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.882968    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53924db-eff9-43c8-8aaa-0aabf2a908f3" containerName="payload-processing"
Oct 30 20:56:59.883033 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.882980    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1d80e7-b0b6-414d-990f-c42d43c48d0f" containerName="payload-pre-processing"
Oct 30 20:56:59.883033 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.882985    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f46cb5-ba9f-47f4-b431-01da37f74f42" containerName="revoke-keys"
Oct 30 20:56:59.887610 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.887590    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:56:59.891701 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.891672    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:56:59.891830 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.891804    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:56:59.896258 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:56:59.896220    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"]
Oct 30 20:57:00.001386 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.001313    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387dbb46-bf78-4764-996f-474903692c9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:00.001386 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.001360    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxhkn\" (UniqueName: \"kubernetes.io/projected/387dbb46-bf78-4764-996f-474903692c9a-kube-api-access-nxhkn\") pod \"maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:00.001578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.001554    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387dbb46-bf78-4764-996f-474903692c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:00.103111 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.103082    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387dbb46-bf78-4764-996f-474903692c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:00.103281 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.103264    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387dbb46-bf78-4764-996f-474903692c9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:00.103331 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.103306    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxhkn\" (UniqueName: \"kubernetes.io/projected/387dbb46-bf78-4764-996f-474903692c9a-kube-api-access-nxhkn\") pod \"maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:00.103808 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.103782    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387dbb46-bf78-4764-996f-474903692c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:00.105749 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.105731    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387dbb46-bf78-4764-996f-474903692c9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:00.111886 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.111851    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxhkn\" (UniqueName: \"kubernetes.io/projected/387dbb46-bf78-4764-996f-474903692c9a-kube-api-access-nxhkn\") pod \"maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:00.206772 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.206742    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:00.499180 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.499149    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"]
Oct 30 20:57:00.715597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:00.715565    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z" event={"ID":"387dbb46-bf78-4764-996f-474903692c9a","Type":"ContainerStarted","Data":"ed3064a533e94ec59608b4f5373d9fe258bf78777b836217476ae9a9ab074e0b"}
Oct 30 20:57:01.725636 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:01.725546    4810 generic.go:356] "Generic (PLEG): container finished" podID="387dbb46-bf78-4764-996f-474903692c9a" containerID="f3eb5d51c9e0b828e3bc12df54a53aa2001c3de18d2450d157881c8910157367" exitCode=0
Oct 30 20:57:01.726011 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:01.725629    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z" event={"ID":"387dbb46-bf78-4764-996f-474903692c9a","Type":"ContainerDied","Data":"f3eb5d51c9e0b828e3bc12df54a53aa2001c3de18d2450d157881c8910157367"}
Oct 30 20:57:02.905498 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:02.905470    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:02.956573 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:02.956529    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/387dbb46-bf78-4764-996f-474903692c9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387dbb46-bf78-4764-996f-474903692c9a-maas-api-xks-ca\") pod \"387dbb46-bf78-4764-996f-474903692c9a\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") "
Oct 30 20:57:02.956802 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:02.956696    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387dbb46-bf78-4764-996f-474903692c9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387dbb46-bf78-4764-996f-474903692c9a-maas-api-openshift-ca\") pod \"387dbb46-bf78-4764-996f-474903692c9a\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") "
Oct 30 20:57:02.956928 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:02.956832    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387dbb46-bf78-4764-996f-474903692c9a-kube-api-access-nxhkn\" (UniqueName: \"kubernetes.io/projected/387dbb46-bf78-4764-996f-474903692c9a-kube-api-access-nxhkn\") pod \"387dbb46-bf78-4764-996f-474903692c9a\" (UID: \"387dbb46-bf78-4764-996f-474903692c9a\") "
Oct 30 20:57:02.960092 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:02.960055    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387dbb46-bf78-4764-996f-474903692c9a-maas-api-openshift-ca" pod "387dbb46-bf78-4764-996f-474903692c9a" (UID: "387dbb46-bf78-4764-996f-474903692c9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:02.964449 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:02.964399    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/387dbb46-bf78-4764-996f-474903692c9a-maas-api-xks-ca" pod "387dbb46-bf78-4764-996f-474903692c9a" (UID: "387dbb46-bf78-4764-996f-474903692c9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:02.967042 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:02.967008    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387dbb46-bf78-4764-996f-474903692c9a-kube-api-access-nxhkn" pod "387dbb46-bf78-4764-996f-474903692c9a" (UID: "387dbb46-bf78-4764-996f-474903692c9a"). InnerVolumeSpecName "kube-api-access-nxhkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:03.058731 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.058645    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxhkn\" (UniqueName: \"kubernetes.io/projected/387dbb46-bf78-4764-996f-474903692c9a-kube-api-access-nxhkn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:03.058731 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.058673    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387dbb46-bf78-4764-996f-474903692c9a-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:03.058731 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.058684    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387dbb46-bf78-4764-996f-474903692c9a-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:03.743946 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.743915    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"
Oct 30 20:57:03.744115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.743936    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z" event={"ID":"387dbb46-bf78-4764-996f-474903692c9a","Type":"ContainerDied","Data":"ed3064a533e94ec59608b4f5373d9fe258bf78777b836217476ae9a9ab074e0b"}
Oct 30 20:57:03.744115 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.743995    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed3064a533e94ec59608b4f5373d9fe258bf78777b836217476ae9a9ab074e0b"
Oct 30 20:57:03.952145 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.952063    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"]
Oct 30 20:57:03.953638 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.953605    4810 cpu_manager.go:414] "containerMap: removing container" podUID="387dbb46-bf78-4764-996f-474903692c9a" containerName="revoke-keys"
Oct 30 20:57:03.953791 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.953778    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387dbb46-bf78-4764-996f-474903692c9a" containerName="revoke-keys"
Oct 30 20:57:03.954106 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.954092    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="387dbb46-bf78-4764-996f-474903692c9a" containerName="revoke-keys"
Oct 30 20:57:03.984226 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.984187    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"]
Oct 30 20:57:03.984423 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.984312    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:03.989067 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.989029    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-351fd1-dockercfg-2q7rw\""
Oct 30 20:57:03.989392 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:03.989377    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-351fd1\""
Oct 30 20:57:04.114892 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.090717    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttz2f\" (UniqueName: \"kubernetes.io/projected/515b4122-b265-4bce-992c-54acf84a640d-kube-api-access-ttz2f\") pod \"payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz\" (UID: \"515b4122-b265-4bce-992c-54acf84a640d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:04.128901 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.126666    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515b4122-b265-4bce-992c-54acf84a640d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz\" (UID: \"515b4122-b265-4bce-992c-54acf84a640d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:04.152880 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.150179    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"]
Oct 30 20:57:04.199882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.196925    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"]
Oct 30 20:57:04.208931 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.205874    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:04.228405 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.228356    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttz2f\" (UniqueName: \"kubernetes.io/projected/515b4122-b265-4bce-992c-54acf84a640d-kube-api-access-ttz2f\") pod \"payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz\" (UID: \"515b4122-b265-4bce-992c-54acf84a640d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:04.228553 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.228432    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515b4122-b265-4bce-992c-54acf84a640d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz\" (UID: \"515b4122-b265-4bce-992c-54acf84a640d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:04.229447 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.229419    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515b4122-b265-4bce-992c-54acf84a640d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz\" (UID: \"515b4122-b265-4bce-992c-54acf84a640d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:04.235569 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.232790    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e019fd-8gjb6"]
Oct 30 20:57:04.241579 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.241545    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttz2f\" (UniqueName: \"kubernetes.io/projected/515b4122-b265-4bce-992c-54acf84a640d-kube-api-access-ttz2f\") pod \"payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz\" (UID: \"515b4122-b265-4bce-992c-54acf84a640d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:04.277237 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.277006    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"]
Oct 30 20:57:04.330193 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.330155    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78429852-14b2-4186-a6b9-2a645849bcd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g\" (UID: \"78429852-14b2-4186-a6b9-2a645849bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:04.330390 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.330227    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zzx\" (UniqueName: \"kubernetes.io/projected/78429852-14b2-4186-a6b9-2a645849bcd5-kube-api-access-v4zzx\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g\" (UID: \"78429852-14b2-4186-a6b9-2a645849bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:04.432130 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.432096    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78429852-14b2-4186-a6b9-2a645849bcd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g\" (UID: \"78429852-14b2-4186-a6b9-2a645849bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:04.432275 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.432167    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zzx\" (UniqueName: \"kubernetes.io/projected/78429852-14b2-4186-a6b9-2a645849bcd5-kube-api-access-v4zzx\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g\" (UID: \"78429852-14b2-4186-a6b9-2a645849bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:04.432811 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.432792    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78429852-14b2-4186-a6b9-2a645849bcd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g\" (UID: \"78429852-14b2-4186-a6b9-2a645849bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:04.434692 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.434674    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:04.444118 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.444090    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zzx\" (UniqueName: \"kubernetes.io/projected/78429852-14b2-4186-a6b9-2a645849bcd5-kube-api-access-v4zzx\") pod \"payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g\" (UID: \"78429852-14b2-4186-a6b9-2a645849bcd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:04.540262 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.540223    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:04.742867 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.742834    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"]
Oct 30 20:57:04.744322 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:57:04.744294    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515b4122_b265_4bce_992c_54acf84a640d.slice/crio-4520db82f2889144654217ea6d361912074328b65b9c1349d88f980dd681505c WatchSource:0}: Error finding container 4520db82f2889144654217ea6d361912074328b65b9c1349d88f980dd681505c: Status 404 returned error can't find the container with id 4520db82f2889144654217ea6d361912074328b65b9c1349d88f980dd681505c
Oct 30 20:57:04.858522 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:04.858491    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"]
Oct 30 20:57:05.663021 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:05.662991    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebc6da4-0e45-4670-92c0-015c1d5e3b17" path="/var/lib/kubelet/pods/7ebc6da4-0e45-4670-92c0-015c1d5e3b17/volumes"
Oct 30 20:57:05.764429 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:05.764279    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g" event={"ID":"78429852-14b2-4186-a6b9-2a645849bcd5","Type":"ContainerStarted","Data":"23af2b38c0c0e649cdce460daf5cce4427ac1893c4b1edccc62df0dcba5d5db1"}
Oct 30 20:57:05.764820 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:05.764795    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:05.764957 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:05.764836    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g" event={"ID":"78429852-14b2-4186-a6b9-2a645849bcd5","Type":"ContainerStarted","Data":"e43a06739f53f8a4aed51a5be0f2d90313084e0549477e459a9e6d21225d4a25"}
Oct 30 20:57:05.766186 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:05.766161    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz" event={"ID":"515b4122-b265-4bce-992c-54acf84a640d","Type":"ContainerStarted","Data":"f8d8156234c259050782788934b41e42194bd5f8c21d047afea58fef8bd48b48"}
Oct 30 20:57:05.766272 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:05.766206    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:05.766272 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:05.766224    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz" event={"ID":"515b4122-b265-4bce-992c-54acf84a640d","Type":"ContainerStarted","Data":"4520db82f2889144654217ea6d361912074328b65b9c1349d88f980dd681505c"}
Oct 30 20:57:05.786265 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:05.786216    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g" podStartSLOduration=1.7862031269999998 podStartE2EDuration="1.786203127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:05.783729475 +0000 UTC m=+1192.275546021" watchObservedRunningTime="2026-10-30 20:57:05.786203127 +0000 UTC m=+1192.278019673"
Oct 30 20:57:07.514732 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.514671    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz" podStartSLOduration=4.514650654 podStartE2EDuration="4.514650654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:05.800443782 +0000 UTC m=+1192.292260328" watchObservedRunningTime="2026-10-30 20:57:07.514650654 +0000 UTC m=+1194.006467208"
Oct 30 20:57:07.517435 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.517413    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"]
Oct 30 20:57:07.523745 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.523209    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:07.528221 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.527835    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:57:07.528324 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.528250    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:57:07.530612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.530590    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"]
Oct 30 20:57:07.677488 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.677456    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-319188-8l5dg\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:07.677652 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.677543    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-319188-8l5dg\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:07.677701 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.677650    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrh6\" (UniqueName: \"kubernetes.io/projected/87361fbb-0d12-4c0b-976f-39b6ad722d9d-kube-api-access-hkrh6\") pod \"maas-api-revoke-keys-e2e-api-a-319188-8l5dg\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:07.778566 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.778488    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrh6\" (UniqueName: \"kubernetes.io/projected/87361fbb-0d12-4c0b-976f-39b6ad722d9d-kube-api-access-hkrh6\") pod \"maas-api-revoke-keys-e2e-api-a-319188-8l5dg\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:07.778773 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.778675    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-319188-8l5dg\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:07.778877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.778785    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-319188-8l5dg\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:07.779510 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.779483    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-319188-8l5dg\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:07.781338 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.781317    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-319188-8l5dg\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:07.795743 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.795724    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrh6\" (UniqueName: \"kubernetes.io/projected/87361fbb-0d12-4c0b-976f-39b6ad722d9d-kube-api-access-hkrh6\") pod \"maas-api-revoke-keys-e2e-api-a-319188-8l5dg\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:07.844552 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:07.844527    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:08.025166 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:08.025134    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"]
Oct 30 20:57:08.040356 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:08.029683    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd796643-rx24v"]
Oct 30 20:57:08.137465 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:57:08.137435    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87361fbb_0d12_4c0b_976f_39b6ad722d9d.slice/crio-3986a6eddeb842dc3acd6e7d764ec1e943a091268d857f2551c195f5caac8097 WatchSource:0}: Error finding container 3986a6eddeb842dc3acd6e7d764ec1e943a091268d857f2551c195f5caac8097: Status 404 returned error can't find the container with id 3986a6eddeb842dc3acd6e7d764ec1e943a091268d857f2551c195f5caac8097
Oct 30 20:57:08.137578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:08.137478    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"]
Oct 30 20:57:08.794471 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:08.794386    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg" event={"ID":"87361fbb-0d12-4c0b-976f-39b6ad722d9d","Type":"ContainerStarted","Data":"cf3989e9f6497b7be22240cf5ac4f34169aea3bf9999000003ee07e113a550d8"}
Oct 30 20:57:08.794471 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:08.794447    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg" event={"ID":"87361fbb-0d12-4c0b-976f-39b6ad722d9d","Type":"ContainerStarted","Data":"3986a6eddeb842dc3acd6e7d764ec1e943a091268d857f2551c195f5caac8097"}
Oct 30 20:57:08.822113 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:08.822055    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg" podStartSLOduration=1.822037378 podStartE2EDuration="1.822037378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:08.815273708 +0000 UTC m=+1195.307090254" watchObservedRunningTime="2026-10-30 20:57:08.822037378 +0000 UTC m=+1195.313853924"
Oct 30 20:57:09.294633 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.294580    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"]
Oct 30 20:57:09.295481 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.294848    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz" podUID="970ab33a-3c05-4f47-bde0-b2717b386da9" containerName="payload-processing" containerID="cri-o://43d5927a3cfa6c93735e81e757984e5937d5eab3b2e321e0d44b1201d7a9ffac" gracePeriod=30
Oct 30 20:57:09.305553 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.305496    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"]
Oct 30 20:57:09.305962 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.305934    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw" podUID="ccf11c95-fc2e-4d94-8cd8-402e743d8113" containerName="payload-pre-processing" containerID="cri-o://051cffc0fc99e0b4c912cb8cf3b9fddf73bca8e0719bf1432439b29c82d3acce" gracePeriod=30
Oct 30 20:57:09.696361 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.695776    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e0a0a8-ddd7-4d22-91cf-e11612132492" path="/var/lib/kubelet/pods/27e0a0a8-ddd7-4d22-91cf-e11612132492/volumes"
Oct 30 20:57:09.819505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.819467    4810 generic.go:356] "Generic (PLEG): container finished" podID="ccf11c95-fc2e-4d94-8cd8-402e743d8113" containerID="051cffc0fc99e0b4c912cb8cf3b9fddf73bca8e0719bf1432439b29c82d3acce" exitCode=0
Oct 30 20:57:09.819987 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.819727    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw" event={"ID":"ccf11c95-fc2e-4d94-8cd8-402e743d8113","Type":"ContainerDied","Data":"051cffc0fc99e0b4c912cb8cf3b9fddf73bca8e0719bf1432439b29c82d3acce"}
Oct 30 20:57:09.819987 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.819777    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw" event={"ID":"ccf11c95-fc2e-4d94-8cd8-402e743d8113","Type":"ContainerDied","Data":"4ea3a757c600642eb9f74c2978023f8a3711d903048201321c04bf60ca45218b"}
Oct 30 20:57:09.819987 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.819799    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ea3a757c600642eb9f74c2978023f8a3711d903048201321c04bf60ca45218b"
Oct 30 20:57:09.822836 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.822809    4810 generic.go:356] "Generic (PLEG): container finished" podID="970ab33a-3c05-4f47-bde0-b2717b386da9" containerID="43d5927a3cfa6c93735e81e757984e5937d5eab3b2e321e0d44b1201d7a9ffac" exitCode=0
Oct 30 20:57:09.822998 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.822939    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz" event={"ID":"970ab33a-3c05-4f47-bde0-b2717b386da9","Type":"ContainerDied","Data":"43d5927a3cfa6c93735e81e757984e5937d5eab3b2e321e0d44b1201d7a9ffac"}
Oct 30 20:57:09.823710 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.823689    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:57:09.825485 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.825462    4810 generic.go:356] "Generic (PLEG): container finished" podID="87361fbb-0d12-4c0b-976f-39b6ad722d9d" containerID="cf3989e9f6497b7be22240cf5ac4f34169aea3bf9999000003ee07e113a550d8" exitCode=0
Oct 30 20:57:09.825575 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.825529    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg" event={"ID":"87361fbb-0d12-4c0b-976f-39b6ad722d9d","Type":"ContainerDied","Data":"cf3989e9f6497b7be22240cf5ac4f34169aea3bf9999000003ee07e113a550d8"}
Oct 30 20:57:09.932175 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.932146    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:57:09.972380 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.971076    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"]
Oct 30 20:57:09.977795 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:09.977767    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e71c9-c2b5n"]
Oct 30 20:57:10.020068 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.020045    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf11c95-fc2e-4d94-8cd8-402e743d8113-kube-api-access-j4h8h\" (UniqueName: \"kubernetes.io/projected/ccf11c95-fc2e-4d94-8cd8-402e743d8113-kube-api-access-j4h8h\") pod \"ccf11c95-fc2e-4d94-8cd8-402e743d8113\" (UID: \"ccf11c95-fc2e-4d94-8cd8-402e743d8113\") "
Oct 30 20:57:10.020192 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.020166    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/970ab33a-3c05-4f47-bde0-b2717b386da9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970ab33a-3c05-4f47-bde0-b2717b386da9-plugins-config-volume\") pod \"970ab33a-3c05-4f47-bde0-b2717b386da9\" (UID: \"970ab33a-3c05-4f47-bde0-b2717b386da9\") "
Oct 30 20:57:10.020355 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.020342    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970ab33a-3c05-4f47-bde0-b2717b386da9-kube-api-access-tn2ns\" (UniqueName: \"kubernetes.io/projected/970ab33a-3c05-4f47-bde0-b2717b386da9-kube-api-access-tn2ns\") pod \"970ab33a-3c05-4f47-bde0-b2717b386da9\" (UID: \"970ab33a-3c05-4f47-bde0-b2717b386da9\") "
Oct 30 20:57:10.020433 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.020365    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf11c95-fc2e-4d94-8cd8-402e743d8113-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf11c95-fc2e-4d94-8cd8-402e743d8113-plugins-config-volume\") pod \"ccf11c95-fc2e-4d94-8cd8-402e743d8113\" (UID: \"ccf11c95-fc2e-4d94-8cd8-402e743d8113\") "
Oct 30 20:57:10.020558 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.020533    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970ab33a-3c05-4f47-bde0-b2717b386da9-plugins-config-volume" pod "970ab33a-3c05-4f47-bde0-b2717b386da9" (UID: "970ab33a-3c05-4f47-bde0-b2717b386da9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:10.020921 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.020843    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf11c95-fc2e-4d94-8cd8-402e743d8113-plugins-config-volume" pod "ccf11c95-fc2e-4d94-8cd8-402e743d8113" (UID: "ccf11c95-fc2e-4d94-8cd8-402e743d8113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:10.021151 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.021133    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970ab33a-3c05-4f47-bde0-b2717b386da9-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:10.021190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.021157    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf11c95-fc2e-4d94-8cd8-402e743d8113-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:10.022337 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.022315    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf11c95-fc2e-4d94-8cd8-402e743d8113-kube-api-access-j4h8h" pod "ccf11c95-fc2e-4d94-8cd8-402e743d8113" (UID: "ccf11c95-fc2e-4d94-8cd8-402e743d8113"). InnerVolumeSpecName "kube-api-access-j4h8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:10.022461 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.022444    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970ab33a-3c05-4f47-bde0-b2717b386da9-kube-api-access-tn2ns" pod "970ab33a-3c05-4f47-bde0-b2717b386da9" (UID: "970ab33a-3c05-4f47-bde0-b2717b386da9"). InnerVolumeSpecName "kube-api-access-tn2ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:10.122467 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.122425    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn2ns\" (UniqueName: \"kubernetes.io/projected/970ab33a-3c05-4f47-bde0-b2717b386da9-kube-api-access-tn2ns\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:10.122467 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.122471    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4h8h\" (UniqueName: \"kubernetes.io/projected/ccf11c95-fc2e-4d94-8cd8-402e743d8113-kube-api-access-j4h8h\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:10.836884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.836842    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"
Oct 30 20:57:10.837397 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.836906    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz" event={"ID":"970ab33a-3c05-4f47-bde0-b2717b386da9","Type":"ContainerDied","Data":"706efba37ccd30439382c3a1f9c12f2bf2c98f36bdbc238f8ffb804f4bf4713b"}
Oct 30 20:57:10.837397 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.836966    4810 scope.go:122] "RemoveContainer" containerID="43d5927a3cfa6c93735e81e757984e5937d5eab3b2e321e0d44b1201d7a9ffac"
Oct 30 20:57:10.837397 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.837236    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"
Oct 30 20:57:10.868020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.867995    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"]
Oct 30 20:57:10.874053 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.874028    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-9599987c4-z8zkz"]
Oct 30 20:57:10.885882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.885844    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"]
Oct 30 20:57:10.890680 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:10.890654    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-5f496fcf6c-mxhtw"]
Oct 30 20:57:11.022027 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.021998    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:11.033626 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.033601    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87361fbb-0d12-4c0b-976f-39b6ad722d9d-kube-api-access-hkrh6\" (UniqueName: \"kubernetes.io/projected/87361fbb-0d12-4c0b-976f-39b6ad722d9d-kube-api-access-hkrh6\") pod \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") "
Oct 30 20:57:11.033735 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.033674    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-xks-ca\") pod \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") "
Oct 30 20:57:11.033779 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.033731    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-openshift-ca\") pod \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\" (UID: \"87361fbb-0d12-4c0b-976f-39b6ad722d9d\") "
Oct 30 20:57:11.034109 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.034085    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-openshift-ca" pod "87361fbb-0d12-4c0b-976f-39b6ad722d9d" (UID: "87361fbb-0d12-4c0b-976f-39b6ad722d9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:11.034335 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.034320    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:11.036194 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.036173    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-xks-ca" pod "87361fbb-0d12-4c0b-976f-39b6ad722d9d" (UID: "87361fbb-0d12-4c0b-976f-39b6ad722d9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:11.036270 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.036199    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87361fbb-0d12-4c0b-976f-39b6ad722d9d-kube-api-access-hkrh6" pod "87361fbb-0d12-4c0b-976f-39b6ad722d9d" (UID: "87361fbb-0d12-4c0b-976f-39b6ad722d9d"). InnerVolumeSpecName "kube-api-access-hkrh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:11.135995 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.135962    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkrh6\" (UniqueName: \"kubernetes.io/projected/87361fbb-0d12-4c0b-976f-39b6ad722d9d-kube-api-access-hkrh6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:11.135995 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.135991    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87361fbb-0d12-4c0b-976f-39b6ad722d9d-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:11.662917 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.662882    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f46cb5-ba9f-47f4-b431-01da37f74f42" path="/var/lib/kubelet/pods/40f46cb5-ba9f-47f4-b431-01da37f74f42/volumes"
Oct 30 20:57:11.663315 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.663301    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="970ab33a-3c05-4f47-bde0-b2717b386da9" path="/var/lib/kubelet/pods/970ab33a-3c05-4f47-bde0-b2717b386da9/volumes"
Oct 30 20:57:11.663658 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.663646    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf11c95-fc2e-4d94-8cd8-402e743d8113" path="/var/lib/kubelet/pods/ccf11c95-fc2e-4d94-8cd8-402e743d8113/volumes"
Oct 30 20:57:11.853926 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.853887    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg" event={"ID":"87361fbb-0d12-4c0b-976f-39b6ad722d9d","Type":"ContainerDied","Data":"3986a6eddeb842dc3acd6e7d764ec1e943a091268d857f2551c195f5caac8097"}
Oct 30 20:57:11.854505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.854483    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3986a6eddeb842dc3acd6e7d764ec1e943a091268d857f2551c195f5caac8097"
Oct 30 20:57:11.855212 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:11.855043    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"
Oct 30 20:57:14.248681 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:14.248655    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:57:14.286264 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:14.286239    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 20:57:14.457151 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:14.457128    4810 scope.go:122] "RemoveContainer" containerID="bb03be2cc9f611cf84e95bbcfe9c2f5025485e92ef36b13b98fc72a805345198"
Oct 30 20:57:14.468045 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:14.468026    4810 scope.go:122] "RemoveContainer" containerID="4da0730e06c09ac3930ce544debcb7360e3088b57eb3dcc5350f09f2bf7da9a6"
Oct 30 20:57:14.515075 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:14.515051    4810 scope.go:122] "RemoveContainer" containerID="39bbfc5f747e3fd273643cf499f32583181c083d2d88d334b542d1bb29c10690"
Oct 30 20:57:14.528538 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:14.528515    4810 scope.go:122] "RemoveContainer" containerID="1eb87ef27bb22d40dbe909addb1161bb6f392c550cb57bfd997cdb6839fc180f"
Oct 30 20:57:15.026302 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:15.026265    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"]
Oct 30 20:57:15.029728 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:15.029696    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f27c36f-kwbsp"]
Oct 30 20:57:15.664096 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:15.664066    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7f6914-8bc7-44a5-b273-2374b8829692" path="/var/lib/kubelet/pods/ea7f6914-8bc7-44a5-b273-2374b8829692/volumes"
Oct 30 20:57:16.776006 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:16.775972    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:16.776540 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:16.776474    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:17.091064 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:17.090975    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:57:20.087269 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.087229    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"]
Oct 30 20:57:20.100388 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.100347    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541db547-8kpgv"]
Oct 30 20:57:20.126211 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.126183    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"]
Oct 30 20:57:20.126826 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.126812    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ccf11c95-fc2e-4d94-8cd8-402e743d8113" containerName="payload-pre-processing"
Oct 30 20:57:20.126891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.126827    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf11c95-fc2e-4d94-8cd8-402e743d8113" containerName="payload-pre-processing"
Oct 30 20:57:20.126891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.126869    4810 cpu_manager.go:414] "containerMap: removing container" podUID="87361fbb-0d12-4c0b-976f-39b6ad722d9d" containerName="revoke-keys"
Oct 30 20:57:20.126891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.126877    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87361fbb-0d12-4c0b-976f-39b6ad722d9d" containerName="revoke-keys"
Oct 30 20:57:20.126991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.126923    4810 cpu_manager.go:414] "containerMap: removing container" podUID="970ab33a-3c05-4f47-bde0-b2717b386da9" containerName="payload-processing"
Oct 30 20:57:20.126991 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.126928    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="970ab33a-3c05-4f47-bde0-b2717b386da9" containerName="payload-processing"
Oct 30 20:57:20.127067 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.127027    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="970ab33a-3c05-4f47-bde0-b2717b386da9" containerName="payload-processing"
Oct 30 20:57:20.127067 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.127046    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf11c95-fc2e-4d94-8cd8-402e743d8113" containerName="payload-pre-processing"
Oct 30 20:57:20.127067 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.127060    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="87361fbb-0d12-4c0b-976f-39b6ad722d9d" containerName="revoke-keys"
Oct 30 20:57:20.132203 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.132183    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.140068 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.140044    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"]
Oct 30 20:57:20.201380 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.201344    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:57:20.201380 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.201366    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:57:20.288812 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.288774    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.288997 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.288898    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27qc6\" (UniqueName: \"kubernetes.io/projected/cc0c1e5b-38a7-463d-ba67-77d27902a75f-kube-api-access-27qc6\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.289195 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.289174    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.390127 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.390096    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.390313 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.390208    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.390313 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.390241    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27qc6\" (UniqueName: \"kubernetes.io/projected/cc0c1e5b-38a7-463d-ba67-77d27902a75f-kube-api-access-27qc6\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.390961 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.390938    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.392598 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.392577    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.400047 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.400017    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27qc6\" (UniqueName: \"kubernetes.io/projected/cc0c1e5b-38a7-463d-ba67-77d27902a75f-kube-api-access-27qc6\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.449204 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.449174    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:20.737371 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.737340    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"]
Oct 30 20:57:20.740422 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:57:20.740392    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0c1e5b_38a7_463d_ba67_77d27902a75f.slice/crio-955320621b8319810b80989b0d0f2d8afa573410b2bc1a637854c41129a8e0d6 WatchSource:0}: Error finding container 955320621b8319810b80989b0d0f2d8afa573410b2bc1a637854c41129a8e0d6: Status 404 returned error can't find the container with id 955320621b8319810b80989b0d0f2d8afa573410b2bc1a637854c41129a8e0d6
Oct 30 20:57:20.927747 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:20.927713    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d" event={"ID":"cc0c1e5b-38a7-463d-ba67-77d27902a75f","Type":"ContainerStarted","Data":"955320621b8319810b80989b0d0f2d8afa573410b2bc1a637854c41129a8e0d6"}
Oct 30 20:57:21.663891 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:21.663846    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c04e81c-8d87-4e08-abc6-ce0146d66535" path="/var/lib/kubelet/pods/8c04e81c-8d87-4e08-abc6-ce0146d66535/volumes"
Oct 30 20:57:21.937075 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:21.937042    4810 generic.go:356] "Generic (PLEG): container finished" podID="cc0c1e5b-38a7-463d-ba67-77d27902a75f" containerID="45a0b36f977abf47b595d9a378b8c0b84f1b64115be8a618ef8c9843ce5667e9" exitCode=0
Oct 30 20:57:21.937249 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:21.937126    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d" event={"ID":"cc0c1e5b-38a7-463d-ba67-77d27902a75f","Type":"ContainerDied","Data":"45a0b36f977abf47b595d9a378b8c0b84f1b64115be8a618ef8c9843ce5667e9"}
Oct 30 20:57:22.024236 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.024204    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"]
Oct 30 20:57:22.027636 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.027614    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69bc17ee-4r46z"]
Oct 30 20:57:22.314786 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.314706    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"]
Oct 30 20:57:22.317997 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.317962    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l" podUID="36fd16f2-d3cc-4b20-9ed6-cf2937d8e261" containerName="payload-processing" containerID="cri-o://c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc" gracePeriod=30
Oct 30 20:57:22.326302 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.326278    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"]
Oct 30 20:57:22.326499 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.326478    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj" podUID="1e3e0257-7094-49a5-b25e-fdc4f8a9c847" containerName="payload-pre-processing" containerID="cri-o://ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b" gracePeriod=30
Oct 30 20:57:22.826947 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.824120    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:57:22.902344 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.902035    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:57:22.945063 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.945032    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-plugins-config-volume\") pod \"1e3e0257-7094-49a5-b25e-fdc4f8a9c847\" (UID: \"1e3e0257-7094-49a5-b25e-fdc4f8a9c847\") "
Oct 30 20:57:22.945351 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.945266    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-kube-api-access-kkvpt\" (UniqueName: \"kubernetes.io/projected/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-kube-api-access-kkvpt\") pod \"1e3e0257-7094-49a5-b25e-fdc4f8a9c847\" (UID: \"1e3e0257-7094-49a5-b25e-fdc4f8a9c847\") "
Oct 30 20:57:22.945573 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.945415    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-plugins-config-volume" pod "1e3e0257-7094-49a5-b25e-fdc4f8a9c847" (UID: "1e3e0257-7094-49a5-b25e-fdc4f8a9c847"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:22.945947 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.945887    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:22.950937 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.950889    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-kube-api-access-kkvpt" pod "1e3e0257-7094-49a5-b25e-fdc4f8a9c847" (UID: "1e3e0257-7094-49a5-b25e-fdc4f8a9c847"). InnerVolumeSpecName "kube-api-access-kkvpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:22.959048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.954017    4810 generic.go:356] "Generic (PLEG): container finished" podID="1e3e0257-7094-49a5-b25e-fdc4f8a9c847" containerID="ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b" exitCode=0
Oct 30 20:57:22.959048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.954185    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj" event={"ID":"1e3e0257-7094-49a5-b25e-fdc4f8a9c847","Type":"ContainerDied","Data":"ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b"}
Oct 30 20:57:22.959048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.954267    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj" event={"ID":"1e3e0257-7094-49a5-b25e-fdc4f8a9c847","Type":"ContainerDied","Data":"d6aff91f1035ac2172b72e8862048aa3cce86b718ac0b8eda8b1d6b49ce19a4f"}
Oct 30 20:57:22.959048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.954296    4810 scope.go:122] "RemoveContainer" containerID="ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b"
Oct 30 20:57:22.959048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.954429    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"
Oct 30 20:57:22.969697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.967942    4810 generic.go:356] "Generic (PLEG): container finished" podID="36fd16f2-d3cc-4b20-9ed6-cf2937d8e261" containerID="c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc" exitCode=0
Oct 30 20:57:22.969697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.968237    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"
Oct 30 20:57:22.969697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.968358    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l" event={"ID":"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261","Type":"ContainerDied","Data":"c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc"}
Oct 30 20:57:22.969697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:22.968400    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l" event={"ID":"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261","Type":"ContainerDied","Data":"8ff45d0a5718c47e2cabbc563eed9a575bcdb5cef44fac5aa594cd3d9955bace"}
Oct 30 20:57:23.040902 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.040827    4810 scope.go:122] "RemoveContainer" containerID="ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b"
Oct 30 20:57:23.043560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.043535    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"]
Oct 30 20:57:23.045089 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:57:23.045061    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b\": container with ID starting with ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b not found: ID does not exist" containerID="ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b"
Oct 30 20:57:23.045196 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.045103    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b"} err="failed to get container status \"ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b\": rpc error: code = NotFound desc = could not find container \"ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b\": container with ID starting with ecb32ecdc07e76fed1692634790d1d0652c406cf5972cc43256a0bb01648f52b not found: ID does not exist"
Oct 30 20:57:23.045196 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.045127    4810 scope.go:122] "RemoveContainer" containerID="c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc"
Oct 30 20:57:23.049710 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.046491    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-kube-api-access-6pdrb\" (UniqueName: \"kubernetes.io/projected/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-kube-api-access-6pdrb\") pod \"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261\" (UID: \"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261\") "
Oct 30 20:57:23.049710 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.046612    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-plugins-config-volume\") pod \"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261\" (UID: \"36fd16f2-d3cc-4b20-9ed6-cf2937d8e261\") "
Oct 30 20:57:23.050451 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.050232    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkvpt\" (UniqueName: \"kubernetes.io/projected/1e3e0257-7094-49a5-b25e-fdc4f8a9c847-kube-api-access-kkvpt\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:23.051005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.050694    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-plugins-config-volume" pod "36fd16f2-d3cc-4b20-9ed6-cf2937d8e261" (UID: "36fd16f2-d3cc-4b20-9ed6-cf2937d8e261"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:23.051005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.050928    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"]
Oct 30 20:57:23.054142 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.054085    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-kube-api-access-6pdrb" pod "36fd16f2-d3cc-4b20-9ed6-cf2937d8e261" (UID: "36fd16f2-d3cc-4b20-9ed6-cf2937d8e261"). InnerVolumeSpecName "kube-api-access-6pdrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:23.056650 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.056597    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-5fb98956c9-brvjj"]
Oct 30 20:57:23.062234 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.062210    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eebcb4-tjz6z"]
Oct 30 20:57:23.081263 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.081119    4810 scope.go:122] "RemoveContainer" containerID="c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc"
Oct 30 20:57:23.090378 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:57:23.090055    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc\": container with ID starting with c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc not found: ID does not exist" containerID="c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc"
Oct 30 20:57:23.090378 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.090097    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc"} err="failed to get container status \"c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc\": rpc error: code = NotFound desc = could not find container \"c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc\": container with ID starting with c779a066963e9878597fb07c0dd546539f835dda5a35a8eb47c3879adb09f4cc not found: ID does not exist"
Oct 30 20:57:23.151540 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.151504    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pdrb\" (UniqueName: \"kubernetes.io/projected/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-kube-api-access-6pdrb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:23.151540 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.151540    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:23.173428 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.173404    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:23.314779 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.314739    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"]
Oct 30 20:57:23.319583 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.319550    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-69df6b4bb4-26v6l"]
Oct 30 20:57:23.353437 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.353347    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0c1e5b-38a7-463d-ba67-77d27902a75f-kube-api-access-27qc6\" (UniqueName: \"kubernetes.io/projected/cc0c1e5b-38a7-463d-ba67-77d27902a75f-kube-api-access-27qc6\") pod \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") "
Oct 30 20:57:23.353437 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.353393    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-xks-ca\") pod \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") "
Oct 30 20:57:23.353683 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.353461    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-openshift-ca\") pod \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\" (UID: \"cc0c1e5b-38a7-463d-ba67-77d27902a75f\") "
Oct 30 20:57:23.353925 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.353890    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-openshift-ca" pod "cc0c1e5b-38a7-463d-ba67-77d27902a75f" (UID: "cc0c1e5b-38a7-463d-ba67-77d27902a75f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:23.354223 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.354203    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:23.355718 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.355682    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0c1e5b-38a7-463d-ba67-77d27902a75f-kube-api-access-27qc6" pod "cc0c1e5b-38a7-463d-ba67-77d27902a75f" (UID: "cc0c1e5b-38a7-463d-ba67-77d27902a75f"). InnerVolumeSpecName "kube-api-access-27qc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:23.355825 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.355797    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-xks-ca" pod "cc0c1e5b-38a7-463d-ba67-77d27902a75f" (UID: "cc0c1e5b-38a7-463d-ba67-77d27902a75f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:23.455730 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.455698    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27qc6\" (UniqueName: \"kubernetes.io/projected/cc0c1e5b-38a7-463d-ba67-77d27902a75f-kube-api-access-27qc6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:23.455730 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.455727    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0c1e5b-38a7-463d-ba67-77d27902a75f-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:23.664091 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.664062    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3e0257-7094-49a5-b25e-fdc4f8a9c847" path="/var/lib/kubelet/pods/1e3e0257-7094-49a5-b25e-fdc4f8a9c847/volumes"
Oct 30 20:57:23.664473 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.664460    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36fd16f2-d3cc-4b20-9ed6-cf2937d8e261" path="/var/lib/kubelet/pods/36fd16f2-d3cc-4b20-9ed6-cf2937d8e261/volumes"
Oct 30 20:57:23.664836 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.664824    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387dbb46-bf78-4764-996f-474903692c9a" path="/var/lib/kubelet/pods/387dbb46-bf78-4764-996f-474903692c9a/volumes"
Oct 30 20:57:23.665520 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.665506    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec789b8-d0e5-45cd-9095-bd988b1d78f4" path="/var/lib/kubelet/pods/7ec789b8-d0e5-45cd-9095-bd988b1d78f4/volumes"
Oct 30 20:57:23.807665 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.807626    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"]
Oct 30 20:57:23.808295 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.808280    4810 cpu_manager.go:414] "containerMap: removing container" podUID="36fd16f2-d3cc-4b20-9ed6-cf2937d8e261" containerName="payload-processing"
Oct 30 20:57:23.808360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.808297    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36fd16f2-d3cc-4b20-9ed6-cf2937d8e261" containerName="payload-processing"
Oct 30 20:57:23.808360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.808331    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1e3e0257-7094-49a5-b25e-fdc4f8a9c847" containerName="payload-pre-processing"
Oct 30 20:57:23.808360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.808337    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3e0257-7094-49a5-b25e-fdc4f8a9c847" containerName="payload-pre-processing"
Oct 30 20:57:23.808360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.808354    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cc0c1e5b-38a7-463d-ba67-77d27902a75f" containerName="revoke-keys"
Oct 30 20:57:23.808360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.808359    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0c1e5b-38a7-463d-ba67-77d27902a75f" containerName="revoke-keys"
Oct 30 20:57:23.808518 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.808469    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3e0257-7094-49a5-b25e-fdc4f8a9c847" containerName="payload-pre-processing"
Oct 30 20:57:23.808518 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.808479    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0c1e5b-38a7-463d-ba67-77d27902a75f" containerName="revoke-keys"
Oct 30 20:57:23.808518 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.808490    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="36fd16f2-d3cc-4b20-9ed6-cf2937d8e261" containerName="payload-processing"
Oct 30 20:57:23.813609 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.813590    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:23.821016 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.820989    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"]
Oct 30 20:57:23.961927 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.961830    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvmb\" (UniqueName: \"kubernetes.io/projected/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-kube-api-access-8xvmb\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:23.962281 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.962107    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:23.962281 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.962139    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:23.979612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.979583    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"
Oct 30 20:57:23.979612 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.979595    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d" event={"ID":"cc0c1e5b-38a7-463d-ba67-77d27902a75f","Type":"ContainerDied","Data":"955320621b8319810b80989b0d0f2d8afa573410b2bc1a637854c41129a8e0d6"}
Oct 30 20:57:23.979846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:23.979646    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="955320621b8319810b80989b0d0f2d8afa573410b2bc1a637854c41129a8e0d6"
Oct 30 20:57:24.063765 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:24.063730    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:24.063952 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:24.063779    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:24.063952 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:24.063926    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvmb\" (UniqueName: \"kubernetes.io/projected/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-kube-api-access-8xvmb\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:24.064502 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:24.064478    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:24.066199 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:24.066177    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:24.073304 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:24.073282    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvmb\" (UniqueName: \"kubernetes.io/projected/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-kube-api-access-8xvmb\") pod \"maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:24.136213 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:24.136179    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:24.424194 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:24.424173    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"]
Oct 30 20:57:24.424667 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:57:24.424645    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49787dd8_1fa9_4e6a_8da5_9fe9ddb6992b.slice/crio-3f58cb2c7aef88103e1dda950e42ada0990e2b1ad8fd4cd7adc0411798085ab3 WatchSource:0}: Error finding container 3f58cb2c7aef88103e1dda950e42ada0990e2b1ad8fd4cd7adc0411798085ab3: Status 404 returned error can't find the container with id 3f58cb2c7aef88103e1dda950e42ada0990e2b1ad8fd4cd7adc0411798085ab3
Oct 30 20:57:24.991895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:24.991762    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5" event={"ID":"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b","Type":"ContainerStarted","Data":"3bf5100589654615c3e7aba904f987cc45bd8ec810a1dd9444011d28e1dfe7a1"}
Oct 30 20:57:24.991895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:24.991819    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5" event={"ID":"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b","Type":"ContainerStarted","Data":"3f58cb2c7aef88103e1dda950e42ada0990e2b1ad8fd4cd7adc0411798085ab3"}
Oct 30 20:57:25.010166 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:25.010121    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5" podStartSLOduration=2.010107175 podStartE2EDuration="2.010107175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:25.008980515 +0000 UTC m=+1211.500797061" watchObservedRunningTime="2026-10-30 20:57:25.010107175 +0000 UTC m=+1211.501923719"
Oct 30 20:57:25.156773 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:57:25.156732    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:57:26.003609 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:26.003571    4810 generic.go:356] "Generic (PLEG): container finished" podID="49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b" containerID="3bf5100589654615c3e7aba904f987cc45bd8ec810a1dd9444011d28e1dfe7a1" exitCode=0
Oct 30 20:57:26.004103 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:26.003660    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5" event={"ID":"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b","Type":"ContainerDied","Data":"3bf5100589654615c3e7aba904f987cc45bd8ec810a1dd9444011d28e1dfe7a1"}
Oct 30 20:57:27.233353 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:27.233324    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:27.261913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:27.261788    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-kube-api-access-8xvmb\" (UniqueName: \"kubernetes.io/projected/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-kube-api-access-8xvmb\") pod \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") "
Oct 30 20:57:27.262057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:27.261954    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-xks-ca\") pod \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") "
Oct 30 20:57:27.262057 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:27.262022    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-openshift-ca\") pod \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\" (UID: \"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b\") "
Oct 30 20:57:27.262624 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:27.262593    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-openshift-ca" pod "49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b" (UID: "49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:27.264221 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:27.264189    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-xks-ca" pod "49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b" (UID: "49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:27.264362 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:27.264340    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-kube-api-access-8xvmb" pod "49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b" (UID: "49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b"). InnerVolumeSpecName "kube-api-access-8xvmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:27.364360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:27.364323    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xvmb\" (UniqueName: \"kubernetes.io/projected/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-kube-api-access-8xvmb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:27.364360 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:27.364359    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:27.364572 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:27.364401    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:28.021169 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:28.021093    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"
Oct 30 20:57:28.021169 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:28.021134    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5" event={"ID":"49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b","Type":"ContainerDied","Data":"3f58cb2c7aef88103e1dda950e42ada0990e2b1ad8fd4cd7adc0411798085ab3"}
Oct 30 20:57:28.021400 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:28.021186    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f58cb2c7aef88103e1dda950e42ada0990e2b1ad8fd4cd7adc0411798085ab3"
Oct 30 20:57:30.883873 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:30.883827    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"]
Oct 30 20:57:30.884936 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:30.884916    4810 cpu_manager.go:414] "containerMap: removing container" podUID="49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b" containerName="revoke-keys"
Oct 30 20:57:30.884994 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:30.884938    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b" containerName="revoke-keys"
Oct 30 20:57:30.885140 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:30.885128    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b" containerName="revoke-keys"
Oct 30 20:57:30.896018 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:30.895997    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:30.897285 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:30.897254    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"]
Oct 30 20:57:30.899540 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:30.899521    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:57:30.900522 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:30.900495    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:57:31.030486 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.030457    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:31.030684 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.030646    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxjhh\" (UniqueName: \"kubernetes.io/projected/4044c714-adbc-40c6-9a54-95db1457487f-kube-api-access-fxjhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:31.030762 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.030691    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:31.128468 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.128434    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"]
Oct 30 20:57:31.128714 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.128688    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp" podUID="7c4ae44c-0ccc-445a-9cad-ef6146dff1a1" containerName="payload-processing" containerID="cri-o://9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7" gracePeriod=30
Oct 30 20:57:31.132456 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.132426    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:31.133027 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.132642    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxjhh\" (UniqueName: \"kubernetes.io/projected/4044c714-adbc-40c6-9a54-95db1457487f-kube-api-access-fxjhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:31.133027 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.132703    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:31.134002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.133535    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:31.135556 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.135531    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:31.191828 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.160943    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxjhh\" (UniqueName: \"kubernetes.io/projected/4044c714-adbc-40c6-9a54-95db1457487f-kube-api-access-fxjhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:31.219794 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.218122    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:31.711341 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.711103    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:57:31.784805 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.783020    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-kube-api-access-txkdw\" (UniqueName: \"kubernetes.io/projected/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-kube-api-access-txkdw\") pod \"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1\" (UID: \"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1\") "
Oct 30 20:57:31.784805 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.783188    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-plugins-config-volume\") pod \"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1\" (UID: \"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1\") "
Oct 30 20:57:31.784805 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.783768    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"]
Oct 30 20:57:31.785075 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.784961    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-plugins-config-volume" pod "7c4ae44c-0ccc-445a-9cad-ef6146dff1a1" (UID: "7c4ae44c-0ccc-445a-9cad-ef6146dff1a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:31.785694 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:57:31.785671    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4044c714_adbc_40c6_9a54_95db1457487f.slice/crio-da908bfe2f0cfc4717ee08cbdbadb9ea274537b8ddd516ff77ec854725566ad6 WatchSource:0}: Error finding container da908bfe2f0cfc4717ee08cbdbadb9ea274537b8ddd516ff77ec854725566ad6: Status 404 returned error can't find the container with id da908bfe2f0cfc4717ee08cbdbadb9ea274537b8ddd516ff77ec854725566ad6
Oct 30 20:57:31.787558 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.787532    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-kube-api-access-txkdw" pod "7c4ae44c-0ccc-445a-9cad-ef6146dff1a1" (UID: "7c4ae44c-0ccc-445a-9cad-ef6146dff1a1"). InnerVolumeSpecName "kube-api-access-txkdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:31.828468 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.828421    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"]
Oct 30 20:57:31.833048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.833029    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c14e2-zxn4d"]
Oct 30 20:57:31.886368 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.886334    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txkdw\" (UniqueName: \"kubernetes.io/projected/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-kube-api-access-txkdw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:31.886368 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:31.886371    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:32.090603 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:32.090503    4810 generic.go:356] "Generic (PLEG): container finished" podID="7c4ae44c-0ccc-445a-9cad-ef6146dff1a1" containerID="9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7" exitCode=0
Oct 30 20:57:32.090603 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:32.090567    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"
Oct 30 20:57:32.090603 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:32.090588    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp" event={"ID":"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1","Type":"ContainerDied","Data":"9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7"}
Oct 30 20:57:32.090945 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:32.090651    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp" event={"ID":"7c4ae44c-0ccc-445a-9cad-ef6146dff1a1","Type":"ContainerDied","Data":"bb81dce5ecedeb6a9075a515a7f8dc048bb87240e66d1233cd454d824ffe2fb8"}
Oct 30 20:57:32.090945 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:32.090685    4810 scope.go:122] "RemoveContainer" containerID="9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7"
Oct 30 20:57:32.092570 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:32.092542    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb" event={"ID":"4044c714-adbc-40c6-9a54-95db1457487f","Type":"ContainerStarted","Data":"da908bfe2f0cfc4717ee08cbdbadb9ea274537b8ddd516ff77ec854725566ad6"}
Oct 30 20:57:32.112007 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:32.111928    4810 scope.go:122] "RemoveContainer" containerID="9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7"
Oct 30 20:57:32.112356 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:57:32.112333    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7\": container with ID starting with 9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7 not found: ID does not exist" containerID="9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7"
Oct 30 20:57:32.112465 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:32.112371    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7"} err="failed to get container status \"9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7\": rpc error: code = NotFound desc = could not find container \"9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7\": container with ID starting with 9da31e2bf6cb75f1b58d47e037fd5e5ea1f26e8c15aa28602dade41b8992e7f7 not found: ID does not exist"
Oct 30 20:57:32.125641 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:32.125614    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"]
Oct 30 20:57:32.132410 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:32.132386    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c14e2-5cb65d4778-8k9lp"]
Oct 30 20:57:33.125932 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:33.123504    4810 generic.go:356] "Generic (PLEG): container finished" podID="4044c714-adbc-40c6-9a54-95db1457487f" containerID="ab5861eaa525bf984dd3979c4e45ca325344bab2cd05b5f60d30358cd74ac5c5" exitCode=0
Oct 30 20:57:33.125932 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:33.123777    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb" event={"ID":"4044c714-adbc-40c6-9a54-95db1457487f","Type":"ContainerDied","Data":"ab5861eaa525bf984dd3979c4e45ca325344bab2cd05b5f60d30358cd74ac5c5"}
Oct 30 20:57:33.411390 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:33.411353    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"]
Oct 30 20:57:33.417707 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:33.416886    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-319188-8l5dg"]
Oct 30 20:57:33.663820 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:33.663751    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c4ae44c-0ccc-445a-9cad-ef6146dff1a1" path="/var/lib/kubelet/pods/7c4ae44c-0ccc-445a-9cad-ef6146dff1a1/volumes"
Oct 30 20:57:33.664274 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:33.664260    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87361fbb-0d12-4c0b-976f-39b6ad722d9d" path="/var/lib/kubelet/pods/87361fbb-0d12-4c0b-976f-39b6ad722d9d/volumes"
Oct 30 20:57:33.664700 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:33.664687    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0c1e5b-38a7-463d-ba67-77d27902a75f" path="/var/lib/kubelet/pods/cc0c1e5b-38a7-463d-ba67-77d27902a75f/volumes"
Oct 30 20:57:34.296640 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:34.296611    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:34.437277 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:34.437186    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-openshift-ca\") pod \"4044c714-adbc-40c6-9a54-95db1457487f\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") "
Oct 30 20:57:34.437453 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:34.437334    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4044c714-adbc-40c6-9a54-95db1457487f-kube-api-access-fxjhh\" (UniqueName: \"kubernetes.io/projected/4044c714-adbc-40c6-9a54-95db1457487f-kube-api-access-fxjhh\") pod \"4044c714-adbc-40c6-9a54-95db1457487f\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") "
Oct 30 20:57:34.437453 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:34.437430    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-xks-ca\") pod \"4044c714-adbc-40c6-9a54-95db1457487f\" (UID: \"4044c714-adbc-40c6-9a54-95db1457487f\") "
Oct 30 20:57:34.437592 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:34.437487    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-openshift-ca" pod "4044c714-adbc-40c6-9a54-95db1457487f" (UID: "4044c714-adbc-40c6-9a54-95db1457487f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:34.437873 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:34.437841    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.440061 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:34.440038    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4044c714-adbc-40c6-9a54-95db1457487f-kube-api-access-fxjhh" pod "4044c714-adbc-40c6-9a54-95db1457487f" (UID: "4044c714-adbc-40c6-9a54-95db1457487f"). InnerVolumeSpecName "kube-api-access-fxjhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:34.440176 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:34.440076    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-xks-ca" pod "4044c714-adbc-40c6-9a54-95db1457487f" (UID: "4044c714-adbc-40c6-9a54-95db1457487f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:34.539308 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:34.539277    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxjhh\" (UniqueName: \"kubernetes.io/projected/4044c714-adbc-40c6-9a54-95db1457487f-kube-api-access-fxjhh\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:34.539308 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:34.539305    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4044c714-adbc-40c6-9a54-95db1457487f-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:35.139703 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:35.139663    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"
Oct 30 20:57:35.139882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:35.139700    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb" event={"ID":"4044c714-adbc-40c6-9a54-95db1457487f","Type":"ContainerDied","Data":"da908bfe2f0cfc4717ee08cbdbadb9ea274537b8ddd516ff77ec854725566ad6"}
Oct 30 20:57:35.139882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:35.139754    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da908bfe2f0cfc4717ee08cbdbadb9ea274537b8ddd516ff77ec854725566ad6"
Oct 30 20:57:35.592187 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:35.589110    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"]
Oct 30 20:57:35.592187 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:35.589447    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln" podUID="9e84eb10-6a53-46ad-a1ec-5df89a4241e6" containerName="payload-processing" containerID="cri-o://9e76b70f372047b572321441702d539b7e11d9ddf2c8785b9a6c2e4a9bd3ba3d" gracePeriod=30
Oct 30 20:57:35.597231 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:35.597205    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"]
Oct 30 20:57:35.597457 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:35.597411    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj" podUID="4572061f-d2e5-4d59-ad02-ef7cc08a016e" containerName="payload-pre-processing" containerID="cri-o://e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1" gracePeriod=30
Oct 30 20:57:35.992803 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:35.992764    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"]
Oct 30 20:57:35.995466 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:35.995436    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c5bf9a-pl2h5"]
Oct 30 20:57:36.038514 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.038478    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"]
Oct 30 20:57:36.048276 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.048239    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bb216e4-sdsd4"]
Oct 30 20:57:36.162075 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.162011    4810 generic.go:356] "Generic (PLEG): container finished" podID="9e84eb10-6a53-46ad-a1ec-5df89a4241e6" containerID="9e76b70f372047b572321441702d539b7e11d9ddf2c8785b9a6c2e4a9bd3ba3d" exitCode=0
Oct 30 20:57:36.162208 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.162186    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln" event={"ID":"9e84eb10-6a53-46ad-a1ec-5df89a4241e6","Type":"ContainerDied","Data":"9e76b70f372047b572321441702d539b7e11d9ddf2c8785b9a6c2e4a9bd3ba3d"}
Oct 30 20:57:36.163466 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.163446    4810 generic.go:356] "Generic (PLEG): container finished" podID="4572061f-d2e5-4d59-ad02-ef7cc08a016e" containerID="e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1" exitCode=0
Oct 30 20:57:36.163697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.163677    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:57:36.164012 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.163988    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj" event={"ID":"4572061f-d2e5-4d59-ad02-ef7cc08a016e","Type":"ContainerDied","Data":"e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1"}
Oct 30 20:57:36.164069 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.164037    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj" event={"ID":"4572061f-d2e5-4d59-ad02-ef7cc08a016e","Type":"ContainerDied","Data":"f0c0af8b0da036fe320736ab90343f803b87963a2a461f87a6480b6346264f75"}
Oct 30 20:57:36.164069 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.164059    4810 scope.go:122] "RemoveContainer" containerID="e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1"
Oct 30 20:57:36.225182 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.225145    4810 scope.go:122] "RemoveContainer" containerID="e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1"
Oct 30 20:57:36.225502 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:57:36.225481    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1\": container with ID starting with e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1 not found: ID does not exist" containerID="e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1"
Oct 30 20:57:36.225605 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.225507    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1"} err="failed to get container status \"e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1\": rpc error: code = NotFound desc = could not find container \"e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1\": container with ID starting with e7404af8ecbdaababd6c73f211835bdf73d05b3430673d05ce219d50218dabd1 not found: ID does not exist"
Oct 30 20:57:36.227184 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.227167    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:57:36.299749 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.299656    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4572061f-d2e5-4d59-ad02-ef7cc08a016e-kube-api-access-67n5b\" (UniqueName: \"kubernetes.io/projected/4572061f-d2e5-4d59-ad02-ef7cc08a016e-kube-api-access-67n5b\") pod \"4572061f-d2e5-4d59-ad02-ef7cc08a016e\" (UID: \"4572061f-d2e5-4d59-ad02-ef7cc08a016e\") "
Oct 30 20:57:36.299943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.299762    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-kube-api-access-zpw7t\" (UniqueName: \"kubernetes.io/projected/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-kube-api-access-zpw7t\") pod \"9e84eb10-6a53-46ad-a1ec-5df89a4241e6\" (UID: \"9e84eb10-6a53-46ad-a1ec-5df89a4241e6\") "
Oct 30 20:57:36.299943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.299802    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-plugins-config-volume\") pod \"9e84eb10-6a53-46ad-a1ec-5df89a4241e6\" (UID: \"9e84eb10-6a53-46ad-a1ec-5df89a4241e6\") "
Oct 30 20:57:36.299943 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.299879    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4572061f-d2e5-4d59-ad02-ef7cc08a016e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4572061f-d2e5-4d59-ad02-ef7cc08a016e-plugins-config-volume\") pod \"4572061f-d2e5-4d59-ad02-ef7cc08a016e\" (UID: \"4572061f-d2e5-4d59-ad02-ef7cc08a016e\") "
Oct 30 20:57:36.300821 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.300789    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4572061f-d2e5-4d59-ad02-ef7cc08a016e-plugins-config-volume" pod "4572061f-d2e5-4d59-ad02-ef7cc08a016e" (UID: "4572061f-d2e5-4d59-ad02-ef7cc08a016e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:36.301780 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.301748    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-plugins-config-volume" pod "9e84eb10-6a53-46ad-a1ec-5df89a4241e6" (UID: "9e84eb10-6a53-46ad-a1ec-5df89a4241e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:36.305103 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.305073    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4572061f-d2e5-4d59-ad02-ef7cc08a016e-kube-api-access-67n5b" pod "4572061f-d2e5-4d59-ad02-ef7cc08a016e" (UID: "4572061f-d2e5-4d59-ad02-ef7cc08a016e"). InnerVolumeSpecName "kube-api-access-67n5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:36.305762 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.305735    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-kube-api-access-zpw7t" pod "9e84eb10-6a53-46ad-a1ec-5df89a4241e6" (UID: "9e84eb10-6a53-46ad-a1ec-5df89a4241e6"). InnerVolumeSpecName "kube-api-access-zpw7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:36.402284 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.402246    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67n5b\" (UniqueName: \"kubernetes.io/projected/4572061f-d2e5-4d59-ad02-ef7cc08a016e-kube-api-access-67n5b\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:36.402284 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.402276    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpw7t\" (UniqueName: \"kubernetes.io/projected/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-kube-api-access-zpw7t\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:36.402284 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.402290    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e84eb10-6a53-46ad-a1ec-5df89a4241e6-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:36.402518 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:36.402304    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4572061f-d2e5-4d59-ad02-ef7cc08a016e-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:37.173422 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.173394    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"
Oct 30 20:57:37.173916 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.173420    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln" event={"ID":"9e84eb10-6a53-46ad-a1ec-5df89a4241e6","Type":"ContainerDied","Data":"4d55050cce1998fcf795ab711529b08a4199e348ba656530b445b0a1a0a9d5b8"}
Oct 30 20:57:37.173916 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.173487    4810 scope.go:122] "RemoveContainer" containerID="9e76b70f372047b572321441702d539b7e11d9ddf2c8785b9a6c2e4a9bd3ba3d"
Oct 30 20:57:37.180960 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.180800    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"
Oct 30 20:57:37.213914 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.213802    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"]
Oct 30 20:57:37.217575 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.217546    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c5bf9a-68bfb9d64d-cvvln"]
Oct 30 20:57:37.227341 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.227303    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"]
Oct 30 20:57:37.230693 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.230671    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-f6dbb66c5-cdbmj"]
Oct 30 20:57:37.663054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.663026    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4572061f-d2e5-4d59-ad02-ef7cc08a016e" path="/var/lib/kubelet/pods/4572061f-d2e5-4d59-ad02-ef7cc08a016e/volumes"
Oct 30 20:57:37.663498 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.663484    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b" path="/var/lib/kubelet/pods/49787dd8-1fa9-4e6a-8da5-9fe9ddb6992b/volumes"
Oct 30 20:57:37.664519 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.664477    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62adfaf2-b44c-475e-91de-91d430e26a2c" path="/var/lib/kubelet/pods/62adfaf2-b44c-475e-91de-91d430e26a2c/volumes"
Oct 30 20:57:37.665233 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:37.665211    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e84eb10-6a53-46ad-a1ec-5df89a4241e6" path="/var/lib/kubelet/pods/9e84eb10-6a53-46ad-a1ec-5df89a4241e6/volumes"
Oct 30 20:57:40.026877 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.025593    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"]
Oct 30 20:57:40.029581 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.029552    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a20adf-c55f2"]
Oct 30 20:57:40.187217 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187186    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"]
Oct 30 20:57:40.187793 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187779    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4044c714-adbc-40c6-9a54-95db1457487f" containerName="revoke-keys"
Oct 30 20:57:40.187832 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187794    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4044c714-adbc-40c6-9a54-95db1457487f" containerName="revoke-keys"
Oct 30 20:57:40.187832 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187815    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4572061f-d2e5-4d59-ad02-ef7cc08a016e" containerName="payload-pre-processing"
Oct 30 20:57:40.187832 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187820    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4572061f-d2e5-4d59-ad02-ef7cc08a016e" containerName="payload-pre-processing"
Oct 30 20:57:40.187963 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187841    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9e84eb10-6a53-46ad-a1ec-5df89a4241e6" containerName="payload-processing"
Oct 30 20:57:40.187963 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187847    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e84eb10-6a53-46ad-a1ec-5df89a4241e6" containerName="payload-processing"
Oct 30 20:57:40.187963 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187872    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7c4ae44c-0ccc-445a-9cad-ef6146dff1a1" containerName="payload-processing"
Oct 30 20:57:40.187963 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187877    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4ae44c-0ccc-445a-9cad-ef6146dff1a1" containerName="payload-processing"
Oct 30 20:57:40.188076 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187971    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4044c714-adbc-40c6-9a54-95db1457487f" containerName="revoke-keys"
Oct 30 20:57:40.188076 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187982    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4572061f-d2e5-4d59-ad02-ef7cc08a016e" containerName="payload-pre-processing"
Oct 30 20:57:40.188076 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187992    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e84eb10-6a53-46ad-a1ec-5df89a4241e6" containerName="payload-processing"
Oct 30 20:57:40.188076 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.187999    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4ae44c-0ccc-445a-9cad-ef6146dff1a1" containerName="payload-processing"
Oct 30 20:57:40.193308 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.193290    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.199396 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.199370    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"]
Oct 30 20:57:40.271035 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.271009    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:57:40.271168 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.271061    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:57:40.277894 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.277830    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdn6p\" (UniqueName: \"kubernetes.io/projected/3b75102f-25e7-4697-b30e-9f933c626984-kube-api-access-pdn6p\") pod \"maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.278004 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.277973    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.278004 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.277996    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.378844 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.378815    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdn6p\" (UniqueName: \"kubernetes.io/projected/3b75102f-25e7-4697-b30e-9f933c626984-kube-api-access-pdn6p\") pod \"maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.379010 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.378908    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.379010 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.378931    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.379672 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.379652    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.381494 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.381470    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.390056 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.390035    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdn6p\" (UniqueName: \"kubernetes.io/projected/3b75102f-25e7-4697-b30e-9f933c626984-kube-api-access-pdn6p\") pod \"maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.509816 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.509786    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:40.795271 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:40.795244    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"]
Oct 30 20:57:40.796284 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:57:40.796260    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b75102f_25e7_4697_b30e_9f933c626984.slice/crio-3c7127054d8a6152d81a1adafcf65423f437eb59138b6c75e19cc10541d69d9f WatchSource:0}: Error finding container 3c7127054d8a6152d81a1adafcf65423f437eb59138b6c75e19cc10541d69d9f: Status 404 returned error can't find the container with id 3c7127054d8a6152d81a1adafcf65423f437eb59138b6c75e19cc10541d69d9f
Oct 30 20:57:41.231164 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:41.231131    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs" event={"ID":"3b75102f-25e7-4697-b30e-9f933c626984","Type":"ContainerStarted","Data":"6eb572b9899b5ef849752b5d915c851a19803f303ba20ff011ef719b64a5ca97"}
Oct 30 20:57:41.231579 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:41.231193    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs" event={"ID":"3b75102f-25e7-4697-b30e-9f933c626984","Type":"ContainerStarted","Data":"3c7127054d8a6152d81a1adafcf65423f437eb59138b6c75e19cc10541d69d9f"}
Oct 30 20:57:41.264659 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:41.264600    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs" podStartSLOduration=1.26458283 podStartE2EDuration="1.26458283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:41.259068595 +0000 UTC m=+1227.750885140" watchObservedRunningTime="2026-10-30 20:57:41.26458283 +0000 UTC m=+1227.756399377"
Oct 30 20:57:41.665349 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:41.665318    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81785d4-5c66-4d32-be6e-d443bc4d911c" path="/var/lib/kubelet/pods/a81785d4-5c66-4d32-be6e-d443bc4d911c/volumes"
Oct 30 20:57:41.809799 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:41.809765    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"]
Oct 30 20:57:41.819978 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:41.819947    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"]
Oct 30 20:57:41.819978 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:41.819969    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:41.911347 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:41.911312    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5452c3-86mk8\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:41.911507 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:41.911442    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5452c3-86mk8\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:41.911580 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:41.911535    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glz9q\" (UniqueName: \"kubernetes.io/projected/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-kube-api-access-glz9q\") pod \"maas-api-revoke-keys-e2e-api-b-5452c3-86mk8\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:42.012912 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:42.012834    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5452c3-86mk8\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:42.013077 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:42.012936    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5452c3-86mk8\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:42.013077 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:42.012993    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glz9q\" (UniqueName: \"kubernetes.io/projected/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-kube-api-access-glz9q\") pod \"maas-api-revoke-keys-e2e-api-b-5452c3-86mk8\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:42.013610 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:42.013588    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5452c3-86mk8\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:42.015204 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:42.015181    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5452c3-86mk8\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:42.021936 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:42.021911    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glz9q\" (UniqueName: \"kubernetes.io/projected/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-kube-api-access-glz9q\") pod \"maas-api-revoke-keys-e2e-api-b-5452c3-86mk8\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:42.155387 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:42.155354    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:42.247800 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:42.247758    4810 generic.go:356] "Generic (PLEG): container finished" podID="3b75102f-25e7-4697-b30e-9f933c626984" containerID="6eb572b9899b5ef849752b5d915c851a19803f303ba20ff011ef719b64a5ca97" exitCode=0
Oct 30 20:57:42.247800 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:42.247794    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs" event={"ID":"3b75102f-25e7-4697-b30e-9f933c626984","Type":"ContainerDied","Data":"6eb572b9899b5ef849752b5d915c851a19803f303ba20ff011ef719b64a5ca97"}
Oct 30 20:57:42.437456 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:42.437422    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"]
Oct 30 20:57:42.441007 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:57:42.440977    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda496eb2f_70c6_4bff_bcd2_1ba1970e434d.slice/crio-dfaa3bcf7ebf196599e1d129accdfd25a76a53fa1baf92bd465194ef9e695528 WatchSource:0}: Error finding container dfaa3bcf7ebf196599e1d129accdfd25a76a53fa1baf92bd465194ef9e695528: Status 404 returned error can't find the container with id dfaa3bcf7ebf196599e1d129accdfd25a76a53fa1baf92bd465194ef9e695528
Oct 30 20:57:43.259726 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.259679    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8" event={"ID":"a496eb2f-70c6-4bff-bcd2-1ba1970e434d","Type":"ContainerStarted","Data":"713033b722894b772bc1435544e322aedf3399293544032266289ca5e3dfddb3"}
Oct 30 20:57:43.260168 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.259765    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8" event={"ID":"a496eb2f-70c6-4bff-bcd2-1ba1970e434d","Type":"ContainerStarted","Data":"dfaa3bcf7ebf196599e1d129accdfd25a76a53fa1baf92bd465194ef9e695528"}
Oct 30 20:57:43.294667 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.294534    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8" podStartSLOduration=2.294509439 podStartE2EDuration="2.294509439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:43.284344647 +0000 UTC m=+1229.776161191" watchObservedRunningTime="2026-10-30 20:57:43.294509439 +0000 UTC m=+1229.786325985"
Oct 30 20:57:43.488983 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.488955    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:43.533008 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.532935    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-openshift-ca\") pod \"3b75102f-25e7-4697-b30e-9f933c626984\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") "
Oct 30 20:57:43.533518 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.533408    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-openshift-ca" pod "3b75102f-25e7-4697-b30e-9f933c626984" (UID: "3b75102f-25e7-4697-b30e-9f933c626984"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:43.533518 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.533509    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-xks-ca\") pod \"3b75102f-25e7-4697-b30e-9f933c626984\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") "
Oct 30 20:57:43.533686 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.533579    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b75102f-25e7-4697-b30e-9f933c626984-kube-api-access-pdn6p\" (UniqueName: \"kubernetes.io/projected/3b75102f-25e7-4697-b30e-9f933c626984-kube-api-access-pdn6p\") pod \"3b75102f-25e7-4697-b30e-9f933c626984\" (UID: \"3b75102f-25e7-4697-b30e-9f933c626984\") "
Oct 30 20:57:43.534226 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.534200    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:43.535649 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.535601    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-xks-ca" pod "3b75102f-25e7-4697-b30e-9f933c626984" (UID: "3b75102f-25e7-4697-b30e-9f933c626984"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:43.542063 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.542035    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b75102f-25e7-4697-b30e-9f933c626984-kube-api-access-pdn6p" pod "3b75102f-25e7-4697-b30e-9f933c626984" (UID: "3b75102f-25e7-4697-b30e-9f933c626984"). InnerVolumeSpecName "kube-api-access-pdn6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:43.640882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.639828    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdn6p\" (UniqueName: \"kubernetes.io/projected/3b75102f-25e7-4697-b30e-9f933c626984-kube-api-access-pdn6p\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:43.640882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:43.639956    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b75102f-25e7-4697-b30e-9f933c626984-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:44.054090 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:44.054056    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"]
Oct 30 20:57:44.072661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:44.072628    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66e0149d-5tc4t"]
Oct 30 20:57:44.278934 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:44.278897    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"
Oct 30 20:57:44.279438 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:44.278899    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs" event={"ID":"3b75102f-25e7-4697-b30e-9f933c626984","Type":"ContainerDied","Data":"3c7127054d8a6152d81a1adafcf65423f437eb59138b6c75e19cc10541d69d9f"}
Oct 30 20:57:44.279438 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:44.279061    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c7127054d8a6152d81a1adafcf65423f437eb59138b6c75e19cc10541d69d9f"
Oct 30 20:57:44.281419 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:44.281363    4810 generic.go:356] "Generic (PLEG): container finished" podID="a496eb2f-70c6-4bff-bcd2-1ba1970e434d" containerID="713033b722894b772bc1435544e322aedf3399293544032266289ca5e3dfddb3" exitCode=0
Oct 30 20:57:44.281597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:44.281556    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8" event={"ID":"a496eb2f-70c6-4bff-bcd2-1ba1970e434d","Type":"ContainerDied","Data":"713033b722894b772bc1435544e322aedf3399293544032266289ca5e3dfddb3"}
Oct 30 20:57:45.486412 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.486378    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:45.606930 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.605841    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-kube-api-access-glz9q\" (UniqueName: \"kubernetes.io/projected/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-kube-api-access-glz9q\") pod \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") "
Oct 30 20:57:45.606930 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.606098    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-xks-ca\") pod \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") "
Oct 30 20:57:45.606930 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.606143    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-openshift-ca\") pod \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\" (UID: \"a496eb2f-70c6-4bff-bcd2-1ba1970e434d\") "
Oct 30 20:57:45.642923 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.636675    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-openshift-ca" pod "a496eb2f-70c6-4bff-bcd2-1ba1970e434d" (UID: "a496eb2f-70c6-4bff-bcd2-1ba1970e434d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:45.675888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.659419    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-xks-ca" pod "a496eb2f-70c6-4bff-bcd2-1ba1970e434d" (UID: "a496eb2f-70c6-4bff-bcd2-1ba1970e434d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:45.675888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.659517    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-kube-api-access-glz9q" pod "a496eb2f-70c6-4bff-bcd2-1ba1970e434d" (UID: "a496eb2f-70c6-4bff-bcd2-1ba1970e434d"). InnerVolumeSpecName "kube-api-access-glz9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:45.714889 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.709953    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glz9q\" (UniqueName: \"kubernetes.io/projected/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-kube-api-access-glz9q\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:45.714889 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.709994    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:45.714889 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.710007    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a496eb2f-70c6-4bff-bcd2-1ba1970e434d-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:45.723125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:45.716027    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02525b8-208a-43e1-baff-acec730e14f4" path="/var/lib/kubelet/pods/d02525b8-208a-43e1-baff-acec730e14f4/volumes"
Oct 30 20:57:46.302632 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:46.302045    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"
Oct 30 20:57:46.302632 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:46.302074    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8" event={"ID":"a496eb2f-70c6-4bff-bcd2-1ba1970e434d","Type":"ContainerDied","Data":"dfaa3bcf7ebf196599e1d129accdfd25a76a53fa1baf92bd465194ef9e695528"}
Oct 30 20:57:46.302632 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:46.302123    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfaa3bcf7ebf196599e1d129accdfd25a76a53fa1baf92bd465194ef9e695528"
Oct 30 20:57:46.713751 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:46.713715    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"]
Oct 30 20:57:46.717281 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:46.717253    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a61855-d2kbb"]
Oct 30 20:57:47.662295 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:47.662261    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4044c714-adbc-40c6-9a54-95db1457487f" path="/var/lib/kubelet/pods/4044c714-adbc-40c6-9a54-95db1457487f/volumes"
Oct 30 20:57:52.037930 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:52.037897    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"]
Oct 30 20:57:52.052312 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:52.052280    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"]
Oct 30 20:57:52.293114 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:52.293036    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:57:55.157157 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:57:55.157121    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:57:56.431436 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:56.431397    4810 generic.go:356] "Generic (PLEG): container finished" podID="23526b46-033a-49da-bbaa-72977ecc7645" containerID="35405428b7bbc6885621445041bfe56650a1cbc9058176c12d6ad8c17b1bdbf1" exitCode=0
Oct 30 20:57:56.431955 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:56.431533    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kl7gz-rqftw" event={"ID":"23526b46-033a-49da-bbaa-72977ecc7645","Type":"ContainerDied","Data":"35405428b7bbc6885621445041bfe56650a1cbc9058176c12d6ad8c17b1bdbf1"}
Oct 30 20:57:57.038769 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.038711    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6" podUID="c6785af5-f24c-43f6-a350-3f79f80e49c3" containerName="payload-processing" containerID="cri-o://88875e12238e72885e01ce0cc0b544791de4fbd3e11d7a9a9b6b4bb2e07afb75" gracePeriod=25
Oct 30 20:57:57.053485 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.053446    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm" podUID="41c4609a-1527-414e-b729-ffdf0c080407" containerName="payload-pre-processing" containerID="cri-o://74093501d0ba9bee38c90a374e0855c677c90116d632b3ca3a68fdb10fed0ac2" gracePeriod=25
Oct 30 20:57:57.448888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.447276    4810 generic.go:356] "Generic (PLEG): container finished" podID="c6785af5-f24c-43f6-a350-3f79f80e49c3" containerID="88875e12238e72885e01ce0cc0b544791de4fbd3e11d7a9a9b6b4bb2e07afb75" exitCode=0
Oct 30 20:57:57.448888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.447400    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6" event={"ID":"c6785af5-f24c-43f6-a350-3f79f80e49c3","Type":"ContainerDied","Data":"88875e12238e72885e01ce0cc0b544791de4fbd3e11d7a9a9b6b4bb2e07afb75"}
Oct 30 20:57:57.448888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.447437    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6" event={"ID":"c6785af5-f24c-43f6-a350-3f79f80e49c3","Type":"ContainerDied","Data":"9368bece9fe55c8947405fc2bc6cace9fe3374ed71e042564c96805dc857f52d"}
Oct 30 20:57:57.448888 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.447452    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9368bece9fe55c8947405fc2bc6cace9fe3374ed71e042564c96805dc857f52d"
Oct 30 20:57:57.449495 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.449041    4810 generic.go:356] "Generic (PLEG): container finished" podID="41c4609a-1527-414e-b729-ffdf0c080407" containerID="74093501d0ba9bee38c90a374e0855c677c90116d632b3ca3a68fdb10fed0ac2" exitCode=0
Oct 30 20:57:57.449495 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.449289    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm" event={"ID":"41c4609a-1527-414e-b729-ffdf0c080407","Type":"ContainerDied","Data":"74093501d0ba9bee38c90a374e0855c677c90116d632b3ca3a68fdb10fed0ac2"}
Oct 30 20:57:57.450197 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.449787    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:57:57.493395 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.493130    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:57:57.537714 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.536135    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6785af5-f24c-43f6-a350-3f79f80e49c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6785af5-f24c-43f6-a350-3f79f80e49c3-plugins-config-volume\") pod \"c6785af5-f24c-43f6-a350-3f79f80e49c3\" (UID: \"c6785af5-f24c-43f6-a350-3f79f80e49c3\") "
Oct 30 20:57:57.537714 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.536429    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6785af5-f24c-43f6-a350-3f79f80e49c3-kube-api-access-745f4\" (UniqueName: \"kubernetes.io/projected/c6785af5-f24c-43f6-a350-3f79f80e49c3-kube-api-access-745f4\") pod \"c6785af5-f24c-43f6-a350-3f79f80e49c3\" (UID: \"c6785af5-f24c-43f6-a350-3f79f80e49c3\") "
Oct 30 20:57:57.538013 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.537875    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6785af5-f24c-43f6-a350-3f79f80e49c3-plugins-config-volume" pod "c6785af5-f24c-43f6-a350-3f79f80e49c3" (UID: "c6785af5-f24c-43f6-a350-3f79f80e49c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:57.545904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.545203    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6785af5-f24c-43f6-a350-3f79f80e49c3-kube-api-access-745f4" pod "c6785af5-f24c-43f6-a350-3f79f80e49c3" (UID: "c6785af5-f24c-43f6-a350-3f79f80e49c3"). InnerVolumeSpecName "kube-api-access-745f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:57.647879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.640294    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c4609a-1527-414e-b729-ffdf0c080407-kube-api-access-v52t7\" (UniqueName: \"kubernetes.io/projected/41c4609a-1527-414e-b729-ffdf0c080407-kube-api-access-v52t7\") pod \"41c4609a-1527-414e-b729-ffdf0c080407\" (UID: \"41c4609a-1527-414e-b729-ffdf0c080407\") "
Oct 30 20:57:57.647879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.640406    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c4609a-1527-414e-b729-ffdf0c080407-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c4609a-1527-414e-b729-ffdf0c080407-plugins-config-volume\") pod \"41c4609a-1527-414e-b729-ffdf0c080407\" (UID: \"41c4609a-1527-414e-b729-ffdf0c080407\") "
Oct 30 20:57:57.647879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.641078    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6785af5-f24c-43f6-a350-3f79f80e49c3-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.647879 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.641103    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-745f4\" (UniqueName: \"kubernetes.io/projected/c6785af5-f24c-43f6-a350-3f79f80e49c3-kube-api-access-745f4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.650097 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.650044    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c4609a-1527-414e-b729-ffdf0c080407-plugins-config-volume" pod "41c4609a-1527-414e-b729-ffdf0c080407" (UID: "41c4609a-1527-414e-b729-ffdf0c080407"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:57.675778 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.675722    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c4609a-1527-414e-b729-ffdf0c080407-kube-api-access-v52t7" pod "41c4609a-1527-414e-b729-ffdf0c080407" (UID: "41c4609a-1527-414e-b729-ffdf0c080407"). InnerVolumeSpecName "kube-api-access-v52t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:57.749316 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.747179    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v52t7\" (UniqueName: \"kubernetes.io/projected/41c4609a-1527-414e-b729-ffdf0c080407-kube-api-access-v52t7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.749316 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.747218    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41c4609a-1527-414e-b729-ffdf0c080407-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.801012 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.800984    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:57:57.949997 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.949905    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23526b46-033a-49da-bbaa-72977ecc7645-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/23526b46-033a-49da-bbaa-72977ecc7645-archives-path\") pod \"23526b46-033a-49da-bbaa-72977ecc7645\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") "
Oct 30 20:57:57.950377 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.950249    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23526b46-033a-49da-bbaa-72977ecc7645-archives-path" pod "23526b46-033a-49da-bbaa-72977ecc7645" (UID: "23526b46-033a-49da-bbaa-72977ecc7645"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:57.950637 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.950617    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-service-ca-bundle\") pod \"23526b46-033a-49da-bbaa-72977ecc7645\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") "
Oct 30 20:57:57.951226 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.951113    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-service-ca-bundle" pod "23526b46-033a-49da-bbaa-72977ecc7645" (UID: "23526b46-033a-49da-bbaa-72977ecc7645"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:57.951421 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.951404    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23526b46-033a-49da-bbaa-72977ecc7645-kube-api-access-bbmm6\" (UniqueName: \"kubernetes.io/projected/23526b46-033a-49da-bbaa-72977ecc7645-kube-api-access-bbmm6\") pod \"23526b46-033a-49da-bbaa-72977ecc7645\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") "
Oct 30 20:57:57.952424 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.952037    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-trusted-ca-bundle\") pod \"23526b46-033a-49da-bbaa-72977ecc7645\" (UID: \"23526b46-033a-49da-bbaa-72977ecc7645\") "
Oct 30 20:57:57.952984 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.952956    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-trusted-ca-bundle" pod "23526b46-033a-49da-bbaa-72977ecc7645" (UID: "23526b46-033a-49da-bbaa-72977ecc7645"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:57.953399 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.953321    4810 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-service-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.953399 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.953345    4810 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23526b46-033a-49da-bbaa-72977ecc7645-trusted-ca-bundle\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.953399 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.953360    4810 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/23526b46-033a-49da-bbaa-72977ecc7645-archives-path\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.961138 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:57.961106    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23526b46-033a-49da-bbaa-72977ecc7645-kube-api-access-bbmm6" pod "23526b46-033a-49da-bbaa-72977ecc7645" (UID: "23526b46-033a-49da-bbaa-72977ecc7645"). InnerVolumeSpecName "kube-api-access-bbmm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:58.055604 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.054317    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"]
Oct 30 20:57:58.055604 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.054559    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz" podUID="515b4122-b265-4bce-992c-54acf84a640d" containerName="payload-processing" containerID="cri-o://f8d8156234c259050782788934b41e42194bd5f8c21d047afea58fef8bd48b48" gracePeriod=30
Oct 30 20:57:58.070696 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.070671    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbmm6\" (UniqueName: \"kubernetes.io/projected/23526b46-033a-49da-bbaa-72977ecc7645-kube-api-access-bbmm6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:58.101750 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.101711    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"]
Oct 30 20:57:58.102125 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.102091    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g" podUID="78429852-14b2-4186-a6b9-2a645849bcd5" containerName="payload-pre-processing" containerID="cri-o://23af2b38c0c0e649cdce460daf5cce4427ac1893c4b1edccc62df0dcba5d5db1" gracePeriod=30
Oct 30 20:57:58.350939 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.350907    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"]
Oct 30 20:57:58.370919 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.365649    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-351fd1-nqnzs"]
Oct 30 20:57:58.583982 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.583723    4810 generic.go:356] "Generic (PLEG): container finished" podID="78429852-14b2-4186-a6b9-2a645849bcd5" containerID="23af2b38c0c0e649cdce460daf5cce4427ac1893c4b1edccc62df0dcba5d5db1" exitCode=0
Oct 30 20:57:58.583982 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.583889    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g" event={"ID":"78429852-14b2-4186-a6b9-2a645849bcd5","Type":"ContainerDied","Data":"23af2b38c0c0e649cdce460daf5cce4427ac1893c4b1edccc62df0dcba5d5db1"}
Oct 30 20:57:58.595609 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.594258    4810 generic.go:356] "Generic (PLEG): container finished" podID="515b4122-b265-4bce-992c-54acf84a640d" containerID="f8d8156234c259050782788934b41e42194bd5f8c21d047afea58fef8bd48b48" exitCode=0
Oct 30 20:57:58.595609 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.594536    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz" event={"ID":"515b4122-b265-4bce-992c-54acf84a640d","Type":"ContainerDied","Data":"f8d8156234c259050782788934b41e42194bd5f8c21d047afea58fef8bd48b48"}
Oct 30 20:57:58.612493 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.612090    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:58.613480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.612807    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"
Oct 30 20:57:58.614589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.613701    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm" event={"ID":"41c4609a-1527-414e-b729-ffdf0c080407","Type":"ContainerDied","Data":"a4e000e71968a26d24556d2601e6889f992c43bf3afba4d94c8205525b49e577"}
Oct 30 20:57:58.614589 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.613827    4810 scope.go:122] "RemoveContainer" containerID="74093501d0ba9bee38c90a374e0855c677c90116d632b3ca3a68fdb10fed0ac2"
Oct 30 20:57:58.625960 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.625384    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"
Oct 30 20:57:58.627507 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.626383    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kl7gz-rqftw" event={"ID":"23526b46-033a-49da-bbaa-72977ecc7645","Type":"ContainerDied","Data":"af7755e6061e223d84e1c4b79bde7012fc2a1d1a9215cdc68c3450c71e490c4d"}
Oct 30 20:57:58.627507 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.626456    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af7755e6061e223d84e1c4b79bde7012fc2a1d1a9215cdc68c3450c71e490c4d"
Oct 30 20:57:58.627507 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.626550    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kl7gz-rqftw"
Oct 30 20:57:58.728228 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.728200    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:58.731476 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.731453    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"]
Oct 30 20:57:58.735191 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.735169    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5452c3-58d4d7f4cf-9dfvm"]
Oct 30 20:57:58.772007 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.771930    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"]
Oct 30 20:57:58.777086 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.777055    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5452c3-798c5694b9-4qwj6"]
Oct 30 20:57:58.813542 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.813505    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515b4122-b265-4bce-992c-54acf84a640d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515b4122-b265-4bce-992c-54acf84a640d-plugins-config-volume\") pod \"515b4122-b265-4bce-992c-54acf84a640d\" (UID: \"515b4122-b265-4bce-992c-54acf84a640d\") "
Oct 30 20:57:58.813699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.813595    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515b4122-b265-4bce-992c-54acf84a640d-kube-api-access-ttz2f\" (UniqueName: \"kubernetes.io/projected/515b4122-b265-4bce-992c-54acf84a640d-kube-api-access-ttz2f\") pod \"515b4122-b265-4bce-992c-54acf84a640d\" (UID: \"515b4122-b265-4bce-992c-54acf84a640d\") "
Oct 30 20:57:58.813699 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.813648    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78429852-14b2-4186-a6b9-2a645849bcd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78429852-14b2-4186-a6b9-2a645849bcd5-plugins-config-volume\") pod \"78429852-14b2-4186-a6b9-2a645849bcd5\" (UID: \"78429852-14b2-4186-a6b9-2a645849bcd5\") "
Oct 30 20:57:58.813837 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.813733    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78429852-14b2-4186-a6b9-2a645849bcd5-kube-api-access-v4zzx\" (UniqueName: \"kubernetes.io/projected/78429852-14b2-4186-a6b9-2a645849bcd5-kube-api-access-v4zzx\") pod \"78429852-14b2-4186-a6b9-2a645849bcd5\" (UID: \"78429852-14b2-4186-a6b9-2a645849bcd5\") "
Oct 30 20:57:58.814760 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.814737    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78429852-14b2-4186-a6b9-2a645849bcd5-plugins-config-volume" pod "78429852-14b2-4186-a6b9-2a645849bcd5" (UID: "78429852-14b2-4186-a6b9-2a645849bcd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:58.815109 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.815090    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515b4122-b265-4bce-992c-54acf84a640d-plugins-config-volume" pod "515b4122-b265-4bce-992c-54acf84a640d" (UID: "515b4122-b265-4bce-992c-54acf84a640d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:58.827480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.827400    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515b4122-b265-4bce-992c-54acf84a640d-kube-api-access-ttz2f" pod "515b4122-b265-4bce-992c-54acf84a640d" (UID: "515b4122-b265-4bce-992c-54acf84a640d"). InnerVolumeSpecName "kube-api-access-ttz2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:58.827480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.827430    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78429852-14b2-4186-a6b9-2a645849bcd5-kube-api-access-v4zzx" pod "78429852-14b2-4186-a6b9-2a645849bcd5" (UID: "78429852-14b2-4186-a6b9-2a645849bcd5"). InnerVolumeSpecName "kube-api-access-v4zzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:58.915517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.915478    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zzx\" (UniqueName: \"kubernetes.io/projected/78429852-14b2-4186-a6b9-2a645849bcd5-kube-api-access-v4zzx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:58.915517 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.915520    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515b4122-b265-4bce-992c-54acf84a640d-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:58.915717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.915537    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttz2f\" (UniqueName: \"kubernetes.io/projected/515b4122-b265-4bce-992c-54acf84a640d-kube-api-access-ttz2f\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:58.915717 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:58.915554    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78429852-14b2-4186-a6b9-2a645849bcd5-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:57:59.635000 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.634970    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"
Oct 30 20:57:59.635486 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.635002    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz" event={"ID":"515b4122-b265-4bce-992c-54acf84a640d","Type":"ContainerDied","Data":"4520db82f2889144654217ea6d361912074328b65b9c1349d88f980dd681505c"}
Oct 30 20:57:59.635486 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.635077    4810 scope.go:122] "RemoveContainer" containerID="f8d8156234c259050782788934b41e42194bd5f8c21d047afea58fef8bd48b48"
Oct 30 20:57:59.637913 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.637888    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g" event={"ID":"78429852-14b2-4186-a6b9-2a645849bcd5","Type":"ContainerDied","Data":"e43a06739f53f8a4aed51a5be0f2d90313084e0549477e459a9e6d21225d4a25"}
Oct 30 20:57:59.638009 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.637952    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"
Oct 30 20:57:59.646221 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.646200    4810 scope.go:122] "RemoveContainer" containerID="23af2b38c0c0e649cdce460daf5cce4427ac1893c4b1edccc62df0dcba5d5db1"
Oct 30 20:57:59.664415 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.664391    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b75102f-25e7-4697-b30e-9f933c626984" path="/var/lib/kubelet/pods/3b75102f-25e7-4697-b30e-9f933c626984/volumes"
Oct 30 20:57:59.664951 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.664935    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c4609a-1527-414e-b729-ffdf0c080407" path="/var/lib/kubelet/pods/41c4609a-1527-414e-b729-ffdf0c080407/volumes"
Oct 30 20:57:59.665323 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.665302    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6785af5-f24c-43f6-a350-3f79f80e49c3" path="/var/lib/kubelet/pods/c6785af5-f24c-43f6-a350-3f79f80e49c3/volumes"
Oct 30 20:57:59.665581 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.665569    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"]
Oct 30 20:57:59.665626 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.665596    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-351fd1-cbb8c8f76-bfjzz"]
Oct 30 20:57:59.680618 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.680599    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"]
Oct 30 20:57:59.684451 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:57:59.684426    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-351fd1-b755dbb5f-fnf4g"]
Oct 30 20:58:01.662448 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:01.662416    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515b4122-b265-4bce-992c-54acf84a640d" path="/var/lib/kubelet/pods/515b4122-b265-4bce-992c-54acf84a640d/volumes"
Oct 30 20:58:01.662840 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:01.662826    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78429852-14b2-4186-a6b9-2a645849bcd5" path="/var/lib/kubelet/pods/78429852-14b2-4186-a6b9-2a645849bcd5/volumes"
Oct 30 20:58:05.328392 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:05.328356    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"]
Oct 30 20:58:05.333184 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:05.333158    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5452c3-86mk8"]
Oct 30 20:58:05.663934 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:05.663904    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a496eb2f-70c6-4bff-bcd2-1ba1970e434d" path="/var/lib/kubelet/pods/a496eb2f-70c6-4bff-bcd2-1ba1970e434d/volumes"
Oct 30 20:58:14.629048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:14.629017    4810 scope.go:122] "RemoveContainer" containerID="0fbd85dad403bc3a6270bb0e712f502e6fac691272c49998ee0a1604f2e66400"
Oct 30 20:58:14.641835 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:14.641534    4810 scope.go:122] "RemoveContainer" containerID="059ba92d7401140d973d7bb894a0096094cc9a2de4b2eb82382009d1f75ce689"
Oct 30 20:58:14.662048 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:14.662019    4810 scope.go:122] "RemoveContainer" containerID="cf90798f8ed97e8e89eb6529c67ccb95587c04b6cc2e8708c2e06c72bfbd3594"
Oct 30 20:58:14.673401 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:14.673353    4810 scope.go:122] "RemoveContainer" containerID="e9c0d04c023eaeabb38487abbfd6b29b72ea73ff5e1d8dad09d39626c1769722"
Oct 30 20:58:14.686643 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:14.686616    4810 scope.go:122] "RemoveContainer" containerID="ba30633cc5ed043733f7c53b42d3d40b7ce79a1f4052f096d6bf8921ca563d67"
Oct 30 20:58:14.696016 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:14.695987    4810 scope.go:122] "RemoveContainer" containerID="68e86cda5c60b087e61506cd83a958d1849277c1945c8d0db9d08ea18092e9df"
Oct 30 20:58:14.708222 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:14.708106    4810 scope.go:122] "RemoveContainer" containerID="dd44819e2507df7128eececf934924c440b3e9855d4f15759d05146c2247b407"
Oct 30 20:58:14.720607 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:14.720587    4810 scope.go:122] "RemoveContainer" containerID="9b5c3a42aa9c3975f599ecdf1be8c81a943e220cacc4d308710ea1992a7d784b"
Oct 30 20:58:25.104971 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:25.104937    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:58:25.159999 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:58:25.159962    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:58:44.789230 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789194    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"]
Oct 30 20:58:44.789798 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789780    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a496eb2f-70c6-4bff-bcd2-1ba1970e434d" containerName="revoke-keys"
Oct 30 20:58:44.789798 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789798    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a496eb2f-70c6-4bff-bcd2-1ba1970e434d" containerName="revoke-keys"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789816    4810 cpu_manager.go:414] "containerMap: removing container" podUID="41c4609a-1527-414e-b729-ffdf0c080407" containerName="payload-pre-processing"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789823    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c4609a-1527-414e-b729-ffdf0c080407" containerName="payload-pre-processing"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789835    4810 cpu_manager.go:414] "containerMap: removing container" podUID="515b4122-b265-4bce-992c-54acf84a640d" containerName="payload-processing"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789839    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515b4122-b265-4bce-992c-54acf84a640d" containerName="payload-processing"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789872    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c6785af5-f24c-43f6-a350-3f79f80e49c3" containerName="payload-processing"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789878    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6785af5-f24c-43f6-a350-3f79f80e49c3" containerName="payload-processing"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789890    4810 cpu_manager.go:414] "containerMap: removing container" podUID="23526b46-033a-49da-bbaa-72977ecc7645" containerName="insights-gathering"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789894    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23526b46-033a-49da-bbaa-72977ecc7645" containerName="insights-gathering"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789909    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3b75102f-25e7-4697-b30e-9f933c626984" containerName="revoke-keys"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789913    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b75102f-25e7-4697-b30e-9f933c626984" containerName="revoke-keys"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789923    4810 cpu_manager.go:414] "containerMap: removing container" podUID="78429852-14b2-4186-a6b9-2a645849bcd5" containerName="payload-pre-processing"
Oct 30 20:58:44.789975 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.789930    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78429852-14b2-4186-a6b9-2a645849bcd5" containerName="payload-pre-processing"
Oct 30 20:58:44.790587 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.790017    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="23526b46-033a-49da-bbaa-72977ecc7645" containerName="insights-gathering"
Oct 30 20:58:44.790587 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.790027    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b75102f-25e7-4697-b30e-9f933c626984" containerName="revoke-keys"
Oct 30 20:58:44.790587 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.790034    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="78429852-14b2-4186-a6b9-2a645849bcd5" containerName="payload-pre-processing"
Oct 30 20:58:44.790587 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.790041    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a496eb2f-70c6-4bff-bcd2-1ba1970e434d" containerName="revoke-keys"
Oct 30 20:58:44.790587 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.790048    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c4609a-1527-414e-b729-ffdf0c080407" containerName="payload-pre-processing"
Oct 30 20:58:44.790587 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.790054    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="515b4122-b265-4bce-992c-54acf84a640d" containerName="payload-processing"
Oct 30 20:58:44.790587 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.790061    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6785af5-f24c-43f6-a350-3f79f80e49c3" containerName="payload-processing"
Oct 30 20:58:44.793731 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.793712    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:58:44.803264 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.803230    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"]
Oct 30 20:58:44.811768 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.806969    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b46fb7-dockercfg-b7g95\""
Oct 30 20:58:44.872134 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.872102    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b46fb7\""
Oct 30 20:58:44.905882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.903300    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rf2m\" (UniqueName: \"kubernetes.io/projected/64ba392d-dca4-4c37-9160-55ff611e7608-kube-api-access-2rf2m\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b\" (UID: \"64ba392d-dca4-4c37-9160-55ff611e7608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:58:44.905882 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:44.903480    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba392d-dca4-4c37-9160-55ff611e7608-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b\" (UID: \"64ba392d-dca4-4c37-9160-55ff611e7608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:58:45.009904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:45.007633    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rf2m\" (UniqueName: \"kubernetes.io/projected/64ba392d-dca4-4c37-9160-55ff611e7608-kube-api-access-2rf2m\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b\" (UID: \"64ba392d-dca4-4c37-9160-55ff611e7608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:58:45.009904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:45.007790    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba392d-dca4-4c37-9160-55ff611e7608-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b\" (UID: \"64ba392d-dca4-4c37-9160-55ff611e7608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:58:45.009904 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:45.008952    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba392d-dca4-4c37-9160-55ff611e7608-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b\" (UID: \"64ba392d-dca4-4c37-9160-55ff611e7608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:58:45.054569 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:45.054479    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rf2m\" (UniqueName: \"kubernetes.io/projected/64ba392d-dca4-4c37-9160-55ff611e7608-kube-api-access-2rf2m\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b\" (UID: \"64ba392d-dca4-4c37-9160-55ff611e7608\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:58:45.127846 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:45.127812    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:58:45.420281 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:45.420256    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"]
Oct 30 20:58:45.422025 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:58:45.421998    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ba392d_dca4_4c37_9160_55ff611e7608.slice/crio-659bf5eb5ced1af6568ae79faa8dfdae40127d0a8b62ceb185e6094d28bffb12 WatchSource:0}: Error finding container 659bf5eb5ced1af6568ae79faa8dfdae40127d0a8b62ceb185e6094d28bffb12: Status 404 returned error can't find the container with id 659bf5eb5ced1af6568ae79faa8dfdae40127d0a8b62ceb185e6094d28bffb12
Oct 30 20:58:46.165597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:46.162461    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b" event={"ID":"64ba392d-dca4-4c37-9160-55ff611e7608","Type":"ContainerStarted","Data":"a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb"}
Oct 30 20:58:46.165597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:46.162532    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b" event={"ID":"64ba392d-dca4-4c37-9160-55ff611e7608","Type":"ContainerStarted","Data":"659bf5eb5ced1af6568ae79faa8dfdae40127d0a8b62ceb185e6094d28bffb12"}
Oct 30 20:58:46.165597 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:46.163587    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:58:46.192682 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:46.192350    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b" podStartSLOduration=2.192329073 podStartE2EDuration="2.192329073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:46.190030634 +0000 UTC m=+1292.681847181" watchObservedRunningTime="2026-10-30 20:58:46.192329073 +0000 UTC m=+1292.684145619"
Oct 30 20:58:55.160059 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:58:55.159961    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:58:58.185244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:58.185219    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:58:59.719929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.719895    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"]
Oct 30 20:58:59.724361 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.724343    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 20:58:59.729482 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.729458    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-955eb7\""
Oct 30 20:58:59.729972 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.729955    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-955eb7-dockercfg-2rr2b\""
Oct 30 20:58:59.734555 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.734531    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"]
Oct 30 20:58:59.749165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.749134    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a7a59d-61be-4175-ad13-d5a5d33fe297-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz\" (UID: \"46a7a59d-61be-4175-ad13-d5a5d33fe297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 20:58:59.749312 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.749293    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgh96\" (UniqueName: \"kubernetes.io/projected/46a7a59d-61be-4175-ad13-d5a5d33fe297-kube-api-access-lgh96\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz\" (UID: \"46a7a59d-61be-4175-ad13-d5a5d33fe297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 20:58:59.858929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.853709    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a7a59d-61be-4175-ad13-d5a5d33fe297-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz\" (UID: \"46a7a59d-61be-4175-ad13-d5a5d33fe297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 20:58:59.858929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.853843    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgh96\" (UniqueName: \"kubernetes.io/projected/46a7a59d-61be-4175-ad13-d5a5d33fe297-kube-api-access-lgh96\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz\" (UID: \"46a7a59d-61be-4175-ad13-d5a5d33fe297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 20:58:59.858929 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.855242    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a7a59d-61be-4175-ad13-d5a5d33fe297-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz\" (UID: \"46a7a59d-61be-4175-ad13-d5a5d33fe297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 20:58:59.907247 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.907214    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"]
Oct 30 20:58:59.914190 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.914164    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 20:58:59.941050 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.941004    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"]
Oct 30 20:58:59.943673 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.943603    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgh96\" (UniqueName: \"kubernetes.io/projected/46a7a59d-61be-4175-ad13-d5a5d33fe297-kube-api-access-lgh96\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz\" (UID: \"46a7a59d-61be-4175-ad13-d5a5d33fe297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 20:58:59.958253 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.958224    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a820c1c-f9f6-4c54-a8eb-423f012d340e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g\" (UID: \"3a820c1c-f9f6-4c54-a8eb-423f012d340e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 20:58:59.958700 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:58:59.958677    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqxd\" (UniqueName: \"kubernetes.io/projected/3a820c1c-f9f6-4c54-a8eb-423f012d340e-kube-api-access-lrqxd\") pod \"payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g\" (UID: \"3a820c1c-f9f6-4c54-a8eb-423f012d340e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 20:59:00.000375 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.000304    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"]
Oct 30 20:59:00.005198 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.005173    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.014130 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.014104    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"]
Oct 30 20:59:00.040950 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.040929    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 20:59:00.059787 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.059750    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf4f6\" (UniqueName: \"kubernetes.io/projected/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-kube-api-access-gf4f6\") pod \"maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.059942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.059808    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a820c1c-f9f6-4c54-a8eb-423f012d340e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g\" (UID: \"3a820c1c-f9f6-4c54-a8eb-423f012d340e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 20:59:00.059942 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.059879    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.060090 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.059990    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqxd\" (UniqueName: \"kubernetes.io/projected/3a820c1c-f9f6-4c54-a8eb-423f012d340e-kube-api-access-lrqxd\") pod \"payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g\" (UID: \"3a820c1c-f9f6-4c54-a8eb-423f012d340e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 20:59:00.060090 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.060043    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.060832 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.060810    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a820c1c-f9f6-4c54-a8eb-423f012d340e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g\" (UID: \"3a820c1c-f9f6-4c54-a8eb-423f012d340e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 20:59:00.078535 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.078493    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqxd\" (UniqueName: \"kubernetes.io/projected/3a820c1c-f9f6-4c54-a8eb-423f012d340e-kube-api-access-lrqxd\") pod \"payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g\" (UID: \"3a820c1c-f9f6-4c54-a8eb-423f012d340e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 20:59:00.092417 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.092394    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:00.092529 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.092504    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:59:00.100314 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.100288    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"]
Oct 30 20:59:00.105669 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.105651    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.114870 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.114629    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"]
Oct 30 20:59:00.161307 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.161277    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf4f6\" (UniqueName: \"kubernetes.io/projected/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-kube-api-access-gf4f6\") pod \"maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.161444 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.161331    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.161444 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.161371    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.161444 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.161434    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjt2\" (UniqueName: \"kubernetes.io/projected/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-kube-api-access-wtjt2\") pod \"maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.161685 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.161483    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.161685 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.161505    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.162756 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.162709    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.167693 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.164108    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.172505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.172462    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf4f6\" (UniqueName: \"kubernetes.io/projected/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-kube-api-access-gf4f6\") pod \"maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.262697 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.262623    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 20:59:00.263054 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.263034    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjt2\" (UniqueName: \"kubernetes.io/projected/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-kube-api-access-wtjt2\") pod \"maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.263178 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.263165    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.263269 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.263259    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.263925 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.263894    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.265516 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.265497    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.271539 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.271516    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjt2\" (UniqueName: \"kubernetes.io/projected/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-kube-api-access-wtjt2\") pod \"maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.336902 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.335639    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:00.363181 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.363154    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"]
Oct 30 20:59:00.364029 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:59:00.364005    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a7a59d_61be_4175_ad13_d5a5d33fe297.slice/crio-9904e53788204093e72f22718e3ff457509f39696c8dda16fcabc3eb6ccecb34 WatchSource:0}: Error finding container 9904e53788204093e72f22718e3ff457509f39696c8dda16fcabc3eb6ccecb34: Status 404 returned error can't find the container with id 9904e53788204093e72f22718e3ff457509f39696c8dda16fcabc3eb6ccecb34
Oct 30 20:59:00.443020 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.442956    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:00.598934 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.598904    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"]
Oct 30 20:59:00.610428 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:59:00.610356    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a820c1c_f9f6_4c54_a8eb_423f012d340e.slice/crio-727e2d992b2a267a26cce42db82d89b6b4ad86207cc82ee25283974ec573d5a0 WatchSource:0}: Error finding container 727e2d992b2a267a26cce42db82d89b6b4ad86207cc82ee25283974ec573d5a0: Status 404 returned error can't find the container with id 727e2d992b2a267a26cce42db82d89b6b4ad86207cc82ee25283974ec573d5a0
Oct 30 20:59:00.645571 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.645541    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"]
Oct 30 20:59:00.673077 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:59:00.673038    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf5ff30_66db_4ddc_b378_0206c2bc7c76.slice/crio-04b875b4ee4aa530a390930a44cd139cb851b275b7c54ce344d6293df83f7c05 WatchSource:0}: Error finding container 04b875b4ee4aa530a390930a44cd139cb851b275b7c54ce344d6293df83f7c05: Status 404 returned error can't find the container with id 04b875b4ee4aa530a390930a44cd139cb851b275b7c54ce344d6293df83f7c05
Oct 30 20:59:00.789467 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:00.789438    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"]
Oct 30 20:59:00.790537 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:59:00.790513    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a43860_8d9f_458e_ba2b_5ff8e3cee8ec.slice/crio-9a6d2e3e5f0c19df8f9f31a1bf786b532573c8ba87afec13f6a5ec9b0c457e41 WatchSource:0}: Error finding container 9a6d2e3e5f0c19df8f9f31a1bf786b532573c8ba87afec13f6a5ec9b0c457e41: Status 404 returned error can't find the container with id 9a6d2e3e5f0c19df8f9f31a1bf786b532573c8ba87afec13f6a5ec9b0c457e41
Oct 30 20:59:01.353443 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.353406    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g" event={"ID":"3a820c1c-f9f6-4c54-a8eb-423f012d340e","Type":"ContainerStarted","Data":"e788547cac81a64dece86ea2f9a33fe0e8df1594d0dd3b6fd3f039a27274f77a"}
Oct 30 20:59:01.353632 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.353466    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g" event={"ID":"3a820c1c-f9f6-4c54-a8eb-423f012d340e","Type":"ContainerStarted","Data":"727e2d992b2a267a26cce42db82d89b6b4ad86207cc82ee25283974ec573d5a0"}
Oct 30 20:59:01.353632 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.353587    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 20:59:01.355333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.355303    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg" event={"ID":"fcf5ff30-66db-4ddc-b378-0206c2bc7c76","Type":"ContainerStarted","Data":"30cecadba1bbbf6efb74f3823d585a5a665327304485f8b8c1b4437d1b4d92a8"}
Oct 30 20:59:01.355465 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.355347    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg" event={"ID":"fcf5ff30-66db-4ddc-b378-0206c2bc7c76","Type":"ContainerStarted","Data":"04b875b4ee4aa530a390930a44cd139cb851b275b7c54ce344d6293df83f7c05"}
Oct 30 20:59:01.357047 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.357010    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb" event={"ID":"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec","Type":"ContainerStarted","Data":"3939c71c94337b74f178de18b330b26464c8fb1a082e3d76992199e6992ba88a"}
Oct 30 20:59:01.357182 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.357051    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb" event={"ID":"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec","Type":"ContainerStarted","Data":"9a6d2e3e5f0c19df8f9f31a1bf786b532573c8ba87afec13f6a5ec9b0c457e41"}
Oct 30 20:59:01.358900 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.358880    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz" event={"ID":"46a7a59d-61be-4175-ad13-d5a5d33fe297","Type":"ContainerStarted","Data":"e7b63ff260f245fd70fdfa24710c702b9554e712feab5c384c907d5712b1c9c6"}
Oct 30 20:59:01.359023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.358922    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 20:59:01.359023 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.358941    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz" event={"ID":"46a7a59d-61be-4175-ad13-d5a5d33fe297","Type":"ContainerStarted","Data":"9904e53788204093e72f22718e3ff457509f39696c8dda16fcabc3eb6ccecb34"}
Oct 30 20:59:01.375895 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.373968    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g" podStartSLOduration=2.373952552 podStartE2EDuration="2.373952552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:01.371029601 +0000 UTC m=+1307.862846146" watchObservedRunningTime="2026-10-30 20:59:01.373952552 +0000 UTC m=+1307.865769098"
Oct 30 20:59:01.388239 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.388193    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg" podStartSLOduration=2.388177302 podStartE2EDuration="2.388177302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:01.386552362 +0000 UTC m=+1307.878368901" watchObservedRunningTime="2026-10-30 20:59:01.388177302 +0000 UTC m=+1307.879993848"
Oct 30 20:59:01.406090 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.406047    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb" podStartSLOduration=1.406036157 podStartE2EDuration="1.406036157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:01.402450029 +0000 UTC m=+1307.894266576" watchObservedRunningTime="2026-10-30 20:59:01.406036157 +0000 UTC m=+1307.897852701"
Oct 30 20:59:01.423093 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:01.423053    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz" podStartSLOduration=2.42304067 podStartE2EDuration="2.42304067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:01.418240946 +0000 UTC m=+1307.910057505" watchObservedRunningTime="2026-10-30 20:59:01.42304067 +0000 UTC m=+1307.914857215"
Oct 30 20:59:02.369112 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:02.369076    4810 generic.go:356] "Generic (PLEG): container finished" podID="fcf5ff30-66db-4ddc-b378-0206c2bc7c76" containerID="30cecadba1bbbf6efb74f3823d585a5a665327304485f8b8c1b4437d1b4d92a8" exitCode=0
Oct 30 20:59:02.369498 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:02.369162    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg" event={"ID":"fcf5ff30-66db-4ddc-b378-0206c2bc7c76","Type":"ContainerDied","Data":"30cecadba1bbbf6efb74f3823d585a5a665327304485f8b8c1b4437d1b4d92a8"}
Oct 30 20:59:02.370739 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:02.370717    4810 generic.go:356] "Generic (PLEG): container finished" podID="13a43860-8d9f-458e-ba2b-5ff8e3cee8ec" containerID="3939c71c94337b74f178de18b330b26464c8fb1a082e3d76992199e6992ba88a" exitCode=0
Oct 30 20:59:02.370853 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:02.370799    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb" event={"ID":"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec","Type":"ContainerDied","Data":"3939c71c94337b74f178de18b330b26464c8fb1a082e3d76992199e6992ba88a"}
Oct 30 20:59:03.603578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.603547    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:03.607436 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.607412    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:03.629389 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.629357    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-xks-ca\") pod \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") "
Oct 30 20:59:03.629561 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.629414    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-kube-api-access-gf4f6\" (UniqueName: \"kubernetes.io/projected/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-kube-api-access-gf4f6\") pod \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") "
Oct 30 20:59:03.629625 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.629582    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-xks-ca\") pod \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") "
Oct 30 20:59:03.629677 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.629621    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-openshift-ca\") pod \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\" (UID: \"fcf5ff30-66db-4ddc-b378-0206c2bc7c76\") "
Oct 30 20:59:03.629735 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.629684    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-openshift-ca\") pod \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") "
Oct 30 20:59:03.629788 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.629751    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-kube-api-access-wtjt2\" (UniqueName: \"kubernetes.io/projected/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-kube-api-access-wtjt2\") pod \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\" (UID: \"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec\") "
Oct 30 20:59:03.630289 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.630263    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-openshift-ca" pod "fcf5ff30-66db-4ddc-b378-0206c2bc7c76" (UID: "fcf5ff30-66db-4ddc-b378-0206c2bc7c76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:03.630451 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.630279    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-openshift-ca" pod "13a43860-8d9f-458e-ba2b-5ff8e3cee8ec" (UID: "13a43860-8d9f-458e-ba2b-5ff8e3cee8ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:03.634302 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.633468    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-xks-ca" pod "fcf5ff30-66db-4ddc-b378-0206c2bc7c76" (UID: "fcf5ff30-66db-4ddc-b378-0206c2bc7c76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:03.634302 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.633548    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-kube-api-access-gf4f6" pod "fcf5ff30-66db-4ddc-b378-0206c2bc7c76" (UID: "fcf5ff30-66db-4ddc-b378-0206c2bc7c76"). InnerVolumeSpecName "kube-api-access-gf4f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:03.634302 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.634246    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-kube-api-access-wtjt2" pod "13a43860-8d9f-458e-ba2b-5ff8e3cee8ec" (UID: "13a43860-8d9f-458e-ba2b-5ff8e3cee8ec"). InnerVolumeSpecName "kube-api-access-wtjt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:03.636906 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.635522    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-xks-ca" pod "13a43860-8d9f-458e-ba2b-5ff8e3cee8ec" (UID: "13a43860-8d9f-458e-ba2b-5ff8e3cee8ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:03.731364 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.731340    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtjt2\" (UniqueName: \"kubernetes.io/projected/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-kube-api-access-wtjt2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:03.731364 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.731364    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:03.731546 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.731375    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf4f6\" (UniqueName: \"kubernetes.io/projected/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-kube-api-access-gf4f6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:03.731546 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.731385    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:03.731546 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.731393    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcf5ff30-66db-4ddc-b378-0206c2bc7c76-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:03.731546 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:03.731402    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:04.390661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:04.390632    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"
Oct 30 20:59:04.390661 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:04.390652    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg" event={"ID":"fcf5ff30-66db-4ddc-b378-0206c2bc7c76","Type":"ContainerDied","Data":"04b875b4ee4aa530a390930a44cd139cb851b275b7c54ce344d6293df83f7c05"}
Oct 30 20:59:04.390967 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:04.390709    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04b875b4ee4aa530a390930a44cd139cb851b275b7c54ce344d6293df83f7c05"
Oct 30 20:59:04.392279 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:04.392261    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"
Oct 30 20:59:04.392403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:04.392310    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb" event={"ID":"13a43860-8d9f-458e-ba2b-5ff8e3cee8ec","Type":"ContainerDied","Data":"9a6d2e3e5f0c19df8f9f31a1bf786b532573c8ba87afec13f6a5ec9b0c457e41"}
Oct 30 20:59:04.392403 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:04.392346    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a6d2e3e5f0c19df8f9f31a1bf786b532573c8ba87afec13f6a5ec9b0c457e41"
Oct 30 20:59:07.781002 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:07.780964    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:59:12.373744 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:12.373713    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 20:59:12.374242 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:12.373791    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 20:59:13.772124 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.772089    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"]
Oct 30 20:59:13.772997 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.772976    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fcf5ff30-66db-4ddc-b378-0206c2bc7c76" containerName="revoke-keys"
Oct 30 20:59:13.773062 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.772999    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf5ff30-66db-4ddc-b378-0206c2bc7c76" containerName="revoke-keys"
Oct 30 20:59:13.773098 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.773066    4810 cpu_manager.go:414] "containerMap: removing container" podUID="13a43860-8d9f-458e-ba2b-5ff8e3cee8ec" containerName="revoke-keys"
Oct 30 20:59:13.773098 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.773075    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a43860-8d9f-458e-ba2b-5ff8e3cee8ec" containerName="revoke-keys"
Oct 30 20:59:13.773236 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.773225    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf5ff30-66db-4ddc-b378-0206c2bc7c76" containerName="revoke-keys"
Oct 30 20:59:13.773269 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.773251    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a43860-8d9f-458e-ba2b-5ff8e3cee8ec" containerName="revoke-keys"
Oct 30 20:59:13.778598 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.778577    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:13.782535 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.782508    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:13.782932 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.782909    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:59:13.784372 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.784336    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"]
Oct 30 20:59:13.922626 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.922572    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:13.922905 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.922848    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5pm\" (UniqueName: \"kubernetes.io/projected/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-kube-api-access-qp5pm\") pod \"maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:13.922994 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:13.922937    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:14.024338 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.024249    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:14.024492 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.024451    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5pm\" (UniqueName: \"kubernetes.io/projected/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-kube-api-access-qp5pm\") pod \"maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:14.024492 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.024485    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:14.024970 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.024944    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:14.026841 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.026817    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:14.033681 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.033660    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5pm\" (UniqueName: \"kubernetes.io/projected/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-kube-api-access-qp5pm\") pod \"maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:14.101391 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.101366    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:14.445682 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.445652    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"]
Oct 30 20:59:14.454898 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:59:14.452621    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d9f836_d1eb_4e5f_aeec_e8aec45ca881.slice/crio-3d2620be51067dd9e600a8c3d805c4714570b9b8f7f79ad59b04193d2c030a18 WatchSource:0}: Error finding container 3d2620be51067dd9e600a8c3d805c4714570b9b8f7f79ad59b04193d2c030a18: Status 404 returned error can't find the container with id 3d2620be51067dd9e600a8c3d805c4714570b9b8f7f79ad59b04193d2c030a18
Oct 30 20:59:14.501213 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.501179    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9" event={"ID":"14d9f836-d1eb-4e5f-aeec-e8aec45ca881","Type":"ContainerStarted","Data":"3d2620be51067dd9e600a8c3d805c4714570b9b8f7f79ad59b04193d2c030a18"}
Oct 30 20:59:14.897096 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.897070    4810 scope.go:122] "RemoveContainer" containerID="604e3797c6f7a0f05952171990e9046f375223e76019f0d9f6934fe714434dd8"
Oct 30 20:59:14.908211 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.908189    4810 scope.go:122] "RemoveContainer" containerID="55f085ef5a6c220e41a2041d2446a6792e573615ab7c3c7929ae4acc39ed0c10"
Oct 30 20:59:14.918948 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.918928    4810 scope.go:122] "RemoveContainer" containerID="882f543620d24ee0fe1bc7e964856a0c32c74c9c15de0099237c484867ca3afa"
Oct 30 20:59:14.928005 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.927964    4810 scope.go:122] "RemoveContainer" containerID="860109773f83a21a54692ab3d9ecf363d24716599708108d179f8d62990dd326"
Oct 30 20:59:14.936457 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:14.936433    4810 scope.go:122] "RemoveContainer" containerID="8144dfe27baffa6e4ecc735aae90bf45b1e39b288ccb923c531c404541430765"
Oct 30 20:59:15.511995 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:15.511962    4810 generic.go:356] "Generic (PLEG): container finished" podID="14d9f836-d1eb-4e5f-aeec-e8aec45ca881" containerID="053ee27fa741e9c66f96cd9b6e591adcd4c898f6b5eceb84295e8147c2245252" exitCode=0
Oct 30 20:59:15.512282 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:15.512266    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9" event={"ID":"14d9f836-d1eb-4e5f-aeec-e8aec45ca881","Type":"ContainerDied","Data":"053ee27fa741e9c66f96cd9b6e591adcd4c898f6b5eceb84295e8147c2245252"}
Oct 30 20:59:16.703176 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:16.703146    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:16.757324 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:16.757287    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-openshift-ca\") pod \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") "
Oct 30 20:59:16.757505 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:16.757377    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-kube-api-access-qp5pm\" (UniqueName: \"kubernetes.io/projected/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-kube-api-access-qp5pm\") pod \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") "
Oct 30 20:59:16.757679 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:16.757650    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-openshift-ca" pod "14d9f836-d1eb-4e5f-aeec-e8aec45ca881" (UID: "14d9f836-d1eb-4e5f-aeec-e8aec45ca881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:16.758030 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:16.757958    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:16.760567 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:16.760541    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-kube-api-access-qp5pm" pod "14d9f836-d1eb-4e5f-aeec-e8aec45ca881" (UID: "14d9f836-d1eb-4e5f-aeec-e8aec45ca881"). InnerVolumeSpecName "kube-api-access-qp5pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:16.858727 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:16.858637    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-xks-ca\") pod \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\" (UID: \"14d9f836-d1eb-4e5f-aeec-e8aec45ca881\") "
Oct 30 20:59:16.859141 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:16.859119    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp5pm\" (UniqueName: \"kubernetes.io/projected/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-kube-api-access-qp5pm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:16.861165 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:16.861132    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-xks-ca" pod "14d9f836-d1eb-4e5f-aeec-e8aec45ca881" (UID: "14d9f836-d1eb-4e5f-aeec-e8aec45ca881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:16.960406 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:16.960366    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d9f836-d1eb-4e5f-aeec-e8aec45ca881-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:17.053318 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:17.053243    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"]
Oct 30 20:59:17.097034 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:17.096997    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3136b91-8kwzv"]
Oct 30 20:59:17.535884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:17.535400    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9" event={"ID":"14d9f836-d1eb-4e5f-aeec-e8aec45ca881","Type":"ContainerDied","Data":"3d2620be51067dd9e600a8c3d805c4714570b9b8f7f79ad59b04193d2c030a18"}
Oct 30 20:59:17.535884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:17.535464    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d2620be51067dd9e600a8c3d805c4714570b9b8f7f79ad59b04193d2c030a18"
Oct 30 20:59:17.535884 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:17.535540    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"
Oct 30 20:59:17.556150 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:17.555990    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"]
Oct 30 20:59:17.683614 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:17.683576    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393e6892-57bd-4867-a88a-0ee89441fd04" path="/var/lib/kubelet/pods/393e6892-57bd-4867-a88a-0ee89441fd04/volumes"
Oct 30 20:59:22.557210 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:22.557143    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b" podUID="64ba392d-dca4-4c37-9160-55ff611e7608" containerName="payload-pre-processing" containerID="cri-o://a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb" gracePeriod=25
Oct 30 20:59:22.846536 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:22.846504    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:59:22.895134 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:22.895104    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ba392d-dca4-4c37-9160-55ff611e7608-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba392d-dca4-4c37-9160-55ff611e7608-plugins-config-volume\") pod \"64ba392d-dca4-4c37-9160-55ff611e7608\" (UID: \"64ba392d-dca4-4c37-9160-55ff611e7608\") "
Oct 30 20:59:22.895320 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:22.895220    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ba392d-dca4-4c37-9160-55ff611e7608-kube-api-access-2rf2m\" (UniqueName: \"kubernetes.io/projected/64ba392d-dca4-4c37-9160-55ff611e7608-kube-api-access-2rf2m\") pod \"64ba392d-dca4-4c37-9160-55ff611e7608\" (UID: \"64ba392d-dca4-4c37-9160-55ff611e7608\") "
Oct 30 20:59:22.895579 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:22.895548    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ba392d-dca4-4c37-9160-55ff611e7608-plugins-config-volume" pod "64ba392d-dca4-4c37-9160-55ff611e7608" (UID: "64ba392d-dca4-4c37-9160-55ff611e7608"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:22.897354 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:22.897324    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ba392d-dca4-4c37-9160-55ff611e7608-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:22.897940 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:22.897910    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ba392d-dca4-4c37-9160-55ff611e7608-kube-api-access-2rf2m" pod "64ba392d-dca4-4c37-9160-55ff611e7608" (UID: "64ba392d-dca4-4c37-9160-55ff611e7608"). InnerVolumeSpecName "kube-api-access-2rf2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:22.998328 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:22.998273    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rf2m\" (UniqueName: \"kubernetes.io/projected/64ba392d-dca4-4c37-9160-55ff611e7608-kube-api-access-2rf2m\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:23.593063 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:23.593015    4810 generic.go:356] "Generic (PLEG): container finished" podID="64ba392d-dca4-4c37-9160-55ff611e7608" containerID="a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb" exitCode=0
Oct 30 20:59:23.593611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:23.593168    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b" event={"ID":"64ba392d-dca4-4c37-9160-55ff611e7608","Type":"ContainerDied","Data":"a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb"}
Oct 30 20:59:23.593611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:23.593282    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b" event={"ID":"64ba392d-dca4-4c37-9160-55ff611e7608","Type":"ContainerDied","Data":"659bf5eb5ced1af6568ae79faa8dfdae40127d0a8b62ceb185e6094d28bffb12"}
Oct 30 20:59:23.593611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:23.593313    4810 scope.go:122] "RemoveContainer" containerID="a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb"
Oct 30 20:59:23.593611 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:23.593480    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"
Oct 30 20:59:23.638516 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:23.638491    4810 scope.go:122] "RemoveContainer" containerID="a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb"
Oct 30 20:59:23.639014 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:59:23.638984    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb\": container with ID starting with a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb not found: ID does not exist" containerID="a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb"
Oct 30 20:59:23.639201 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:23.639028    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb"} err="failed to get container status \"a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb\": rpc error: code = NotFound desc = could not find container \"a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb\": container with ID starting with a57e114dd02547113574ad9bf2866e7e82003e41f5202f44cb26689fb5d7fbfb not found: ID does not exist"
Oct 30 20:59:23.668993 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:23.668962    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"]
Oct 30 20:59:23.671680 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:23.671646    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5c665fd7ff-rh66b"]
Oct 30 20:59:24.227533 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:24.227497    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"]
Oct 30 20:59:24.231905 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:24.231870    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2b7ca7-xqghb"]
Oct 30 20:59:25.155643 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:59:25.155609    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 20:59:25.665111 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:25.665073    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a43860-8d9f-458e-ba2b-5ff8e3cee8ec" path="/var/lib/kubelet/pods/13a43860-8d9f-458e-ba2b-5ff8e3cee8ec/volumes"
Oct 30 20:59:25.665677 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:25.665656    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ba392d-dca4-4c37-9160-55ff611e7608" path="/var/lib/kubelet/pods/64ba392d-dca4-4c37-9160-55ff611e7608/volumes"
Oct 30 20:59:26.944955 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:26.944919    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"]
Oct 30 20:59:26.947912 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:26.947888    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-38c5ca-9bnxg"]
Oct 30 20:59:27.662238 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:27.662208    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcf5ff30-66db-4ddc-b378-0206c2bc7c76" path="/var/lib/kubelet/pods/fcf5ff30-66db-4ddc-b378-0206c2bc7c76/volumes"
Oct 30 20:59:32.374834 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.374792    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"]
Oct 30 20:59:32.376209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.376183    4810 cpu_manager.go:414] "containerMap: removing container" podUID="14d9f836-d1eb-4e5f-aeec-e8aec45ca881" containerName="revoke-keys"
Oct 30 20:59:32.376209 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.376208    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d9f836-d1eb-4e5f-aeec-e8aec45ca881" containerName="revoke-keys"
Oct 30 20:59:32.376414 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.376278    4810 cpu_manager.go:414] "containerMap: removing container" podUID="64ba392d-dca4-4c37-9160-55ff611e7608" containerName="payload-pre-processing"
Oct 30 20:59:32.376414 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.376288    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ba392d-dca4-4c37-9160-55ff611e7608" containerName="payload-pre-processing"
Oct 30 20:59:32.376578 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.376562    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d9f836-d1eb-4e5f-aeec-e8aec45ca881" containerName="revoke-keys"
Oct 30 20:59:32.376657 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.376593    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ba392d-dca4-4c37-9160-55ff611e7608" containerName="payload-pre-processing"
Oct 30 20:59:32.384621 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.384595    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.388553 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.388529    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"]
Oct 30 20:59:32.404721 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.404699    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:32.404935 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.404788    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:59:32.456819 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.456794    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.456959 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.456840    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.457090 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.457071    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b99rf\" (UniqueName: \"kubernetes.io/projected/eb887446-5bc9-4fd7-b6a8-5052ad159859-kube-api-access-b99rf\") pod \"maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.558469 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.558434    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99rf\" (UniqueName: \"kubernetes.io/projected/eb887446-5bc9-4fd7-b6a8-5052ad159859-kube-api-access-b99rf\") pod \"maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.558637 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.558609    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.558704 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.558658    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.559398 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.559372    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.561029 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.561006    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.567333 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.567306    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99rf\" (UniqueName: \"kubernetes.io/projected/eb887446-5bc9-4fd7-b6a8-5052ad159859-kube-api-access-b99rf\") pod \"maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.705549 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.705516    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:32.988742 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:59:32.988713    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb887446_5bc9_4fd7_b6a8_5052ad159859.slice/crio-a2eceb41fb3fef57fb96cddd8d0578faca69c87e75ea8085d025038366203df1 WatchSource:0}: Error finding container a2eceb41fb3fef57fb96cddd8d0578faca69c87e75ea8085d025038366203df1: Status 404 returned error can't find the container with id a2eceb41fb3fef57fb96cddd8d0578faca69c87e75ea8085d025038366203df1
Oct 30 20:59:32.989034 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:32.989005    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"]
Oct 30 20:59:33.030933 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:33.030907    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"]
Oct 30 20:59:33.037763 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:33.037736    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fe1ea85-zxrk2"]
Oct 30 20:59:33.663524 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:33.663453    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33" path="/var/lib/kubelet/pods/bd7be06b-7f5a-43e2-9e0c-2b9a8ec46a33/volumes"
Oct 30 20:59:33.750492 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:33.750454    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965" event={"ID":"eb887446-5bc9-4fd7-b6a8-5052ad159859","Type":"ContainerStarted","Data":"bef0e1dd16d15adfc90cd4f0d5ab1d622342fd0c7c0b38647a6e7f4b2e0ce838"}
Oct 30 20:59:33.750695 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:33.750518    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965" event={"ID":"eb887446-5bc9-4fd7-b6a8-5052ad159859","Type":"ContainerStarted","Data":"a2eceb41fb3fef57fb96cddd8d0578faca69c87e75ea8085d025038366203df1"}
Oct 30 20:59:33.773393 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:33.773304    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965" podStartSLOduration=1.7732843790000001 podStartE2EDuration="1.773284379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:33.771988473 +0000 UTC m=+1340.263805037" watchObservedRunningTime="2026-10-30 20:59:33.773284379 +0000 UTC m=+1340.265100940"
Oct 30 20:59:34.777798 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:34.777754    4810 generic.go:356] "Generic (PLEG): container finished" podID="eb887446-5bc9-4fd7-b6a8-5052ad159859" containerID="bef0e1dd16d15adfc90cd4f0d5ab1d622342fd0c7c0b38647a6e7f4b2e0ce838" exitCode=0
Oct 30 20:59:34.778249 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:34.778034    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965" event={"ID":"eb887446-5bc9-4fd7-b6a8-5052ad159859","Type":"ContainerDied","Data":"bef0e1dd16d15adfc90cd4f0d5ab1d622342fd0c7c0b38647a6e7f4b2e0ce838"}
Oct 30 20:59:35.984997 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:35.984968    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:36.116836 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.116762    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-openshift-ca\") pod \"eb887446-5bc9-4fd7-b6a8-5052ad159859\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") "
Oct 30 20:59:36.116836 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.116826    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb887446-5bc9-4fd7-b6a8-5052ad159859-kube-api-access-b99rf\" (UniqueName: \"kubernetes.io/projected/eb887446-5bc9-4fd7-b6a8-5052ad159859-kube-api-access-b99rf\") pod \"eb887446-5bc9-4fd7-b6a8-5052ad159859\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") "
Oct 30 20:59:36.117041 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.116900    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-xks-ca\") pod \"eb887446-5bc9-4fd7-b6a8-5052ad159859\" (UID: \"eb887446-5bc9-4fd7-b6a8-5052ad159859\") "
Oct 30 20:59:36.118893 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.117514    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-openshift-ca" pod "eb887446-5bc9-4fd7-b6a8-5052ad159859" (UID: "eb887446-5bc9-4fd7-b6a8-5052ad159859"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:36.121137 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.119901    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:36.130396 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.130363    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-xks-ca" pod "eb887446-5bc9-4fd7-b6a8-5052ad159859" (UID: "eb887446-5bc9-4fd7-b6a8-5052ad159859"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:36.130534 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.130479    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb887446-5bc9-4fd7-b6a8-5052ad159859-kube-api-access-b99rf" pod "eb887446-5bc9-4fd7-b6a8-5052ad159859" (UID: "eb887446-5bc9-4fd7-b6a8-5052ad159859"). InnerVolumeSpecName "kube-api-access-b99rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:36.221187 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.221155    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b99rf\" (UniqueName: \"kubernetes.io/projected/eb887446-5bc9-4fd7-b6a8-5052ad159859-kube-api-access-b99rf\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:36.221187 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.221185    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb887446-5bc9-4fd7-b6a8-5052ad159859-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:36.804130 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.804087    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965" event={"ID":"eb887446-5bc9-4fd7-b6a8-5052ad159859","Type":"ContainerDied","Data":"a2eceb41fb3fef57fb96cddd8d0578faca69c87e75ea8085d025038366203df1"}
Oct 30 20:59:36.804335 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.804158    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2eceb41fb3fef57fb96cddd8d0578faca69c87e75ea8085d025038366203df1"
Oct 30 20:59:36.804335 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:36.804099    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"
Oct 30 20:59:38.483560 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:38.483417    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"]
Oct 30 20:59:38.488436 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:38.488408    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c72dc9-9spz9"]
Oct 30 20:59:39.662728 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:39.662694    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d9f836-d1eb-4e5f-aeec-e8aec45ca881" path="/var/lib/kubelet/pods/14d9f836-d1eb-4e5f-aeec-e8aec45ca881/volumes"
Oct 30 20:59:40.070647 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.070579    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"]
Oct 30 20:59:40.071214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.071197    4810 cpu_manager.go:414] "containerMap: removing container" podUID="eb887446-5bc9-4fd7-b6a8-5052ad159859" containerName="revoke-keys"
Oct 30 20:59:40.071214 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.071214    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb887446-5bc9-4fd7-b6a8-5052ad159859" containerName="revoke-keys"
Oct 30 20:59:40.071400 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.071352    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb887446-5bc9-4fd7-b6a8-5052ad159859" containerName="revoke-keys"
Oct 30 20:59:40.076017 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.076000    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.083180 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.083157    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:40.083547 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.083528    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:59:40.086892 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.086843    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"]
Oct 30 20:59:40.202244 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.202214    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.202413 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.202273    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdgw\" (UniqueName: \"kubernetes.io/projected/c5e4ca75-63ad-4823-a138-69e27c6403f0-kube-api-access-8zdgw\") pod \"maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.202485 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.202437    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.304107 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.304072    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.304266 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.304163    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdgw\" (UniqueName: \"kubernetes.io/projected/c5e4ca75-63ad-4823-a138-69e27c6403f0-kube-api-access-8zdgw\") pod \"maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.304346 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.304282    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.304977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.304950    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.306525 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.306505    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.313107 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.313080    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdgw\" (UniqueName: \"kubernetes.io/projected/c5e4ca75-63ad-4823-a138-69e27c6403f0-kube-api-access-8zdgw\") pod \"maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.414260 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.414226    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:40.702129 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.701906    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"]
Oct 30 20:59:40.704212 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:59:40.704180    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e4ca75_63ad_4823_a138_69e27c6403f0.slice/crio-ff7673b3c27c03aaf65757bc20ed73eca528511ffaea91f6486861a59b3a667c WatchSource:0}: Error finding container ff7673b3c27c03aaf65757bc20ed73eca528511ffaea91f6486861a59b3a667c: Status 404 returned error can't find the container with id ff7673b3c27c03aaf65757bc20ed73eca528511ffaea91f6486861a59b3a667c
Oct 30 20:59:40.836941 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:40.836913    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc" event={"ID":"c5e4ca75-63ad-4823-a138-69e27c6403f0","Type":"ContainerStarted","Data":"ff7673b3c27c03aaf65757bc20ed73eca528511ffaea91f6486861a59b3a667c"}
Oct 30 20:59:41.844836 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:41.844757    4810 generic.go:356] "Generic (PLEG): container finished" podID="c5e4ca75-63ad-4823-a138-69e27c6403f0" containerID="1a66f6e042e5ad12f3e1658a5f7688e508a7160239764fa6c07bfaef5e1e5b25" exitCode=0
Oct 30 20:59:41.845248 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:41.844838    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc" event={"ID":"c5e4ca75-63ad-4823-a138-69e27c6403f0","Type":"ContainerDied","Data":"1a66f6e042e5ad12f3e1658a5f7688e508a7160239764fa6c07bfaef5e1e5b25"}
Oct 30 20:59:43.113723 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.113661    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:43.249712 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.249683    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-xks-ca\") pod \"c5e4ca75-63ad-4823-a138-69e27c6403f0\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") "
Oct 30 20:59:43.249919 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.249721    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e4ca75-63ad-4823-a138-69e27c6403f0-kube-api-access-8zdgw\" (UniqueName: \"kubernetes.io/projected/c5e4ca75-63ad-4823-a138-69e27c6403f0-kube-api-access-8zdgw\") pod \"c5e4ca75-63ad-4823-a138-69e27c6403f0\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") "
Oct 30 20:59:43.249919 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.249887    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-openshift-ca\") pod \"c5e4ca75-63ad-4823-a138-69e27c6403f0\" (UID: \"c5e4ca75-63ad-4823-a138-69e27c6403f0\") "
Oct 30 20:59:43.250255 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.250229    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-openshift-ca" pod "c5e4ca75-63ad-4823-a138-69e27c6403f0" (UID: "c5e4ca75-63ad-4823-a138-69e27c6403f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:43.252049 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.252023    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-xks-ca" pod "c5e4ca75-63ad-4823-a138-69e27c6403f0" (UID: "c5e4ca75-63ad-4823-a138-69e27c6403f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:43.252137 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.252077    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e4ca75-63ad-4823-a138-69e27c6403f0-kube-api-access-8zdgw" pod "c5e4ca75-63ad-4823-a138-69e27c6403f0" (UID: "c5e4ca75-63ad-4823-a138-69e27c6403f0"). InnerVolumeSpecName "kube-api-access-8zdgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:43.351532 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.351505    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:43.351532 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.351529    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdgw\" (UniqueName: \"kubernetes.io/projected/c5e4ca75-63ad-4823-a138-69e27c6403f0-kube-api-access-8zdgw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:43.351691 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.351541    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e4ca75-63ad-4823-a138-69e27c6403f0-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:43.860767 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.860724    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"
Oct 30 20:59:43.860767 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.860749    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc" event={"ID":"c5e4ca75-63ad-4823-a138-69e27c6403f0","Type":"ContainerDied","Data":"ff7673b3c27c03aaf65757bc20ed73eca528511ffaea91f6486861a59b3a667c"}
Oct 30 20:59:43.861019 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:43.860808    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff7673b3c27c03aaf65757bc20ed73eca528511ffaea91f6486861a59b3a667c"
Oct 30 20:59:44.843624 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:44.843569    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 20:59:46.179807 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.179766    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"]
Oct 30 20:59:46.180472 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.180455    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c5e4ca75-63ad-4823-a138-69e27c6403f0" containerName="revoke-keys"
Oct 30 20:59:46.180521 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.180475    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e4ca75-63ad-4823-a138-69e27c6403f0" containerName="revoke-keys"
Oct 30 20:59:46.180600 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.180590    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e4ca75-63ad-4823-a138-69e27c6403f0" containerName="revoke-keys"
Oct 30 20:59:46.185316 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.185296    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.192651 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.192628    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:46.196809 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.196790    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 20:59:46.197424 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.197397    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"]
Oct 30 20:59:46.319247 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.319216    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnznw\" (UniqueName: \"kubernetes.io/projected/f728965a-371e-4587-96cf-91d70bf0d9e9-kube-api-access-xnznw\") pod \"maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.319398 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.319349    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.319526 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.319511    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.420755 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.420716    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.420977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.420784    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnznw\" (UniqueName: \"kubernetes.io/projected/f728965a-371e-4587-96cf-91d70bf0d9e9-kube-api-access-xnznw\") pod \"maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.420977 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.420849    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.421577 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.421552    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.423222 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.423200    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.428790 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.428771    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnznw\" (UniqueName: \"kubernetes.io/projected/f728965a-371e-4587-96cf-91d70bf0d9e9-kube-api-access-xnznw\") pod \"maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.507470 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.507375    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:46.788738 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.788712    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"]
Oct 30 20:59:46.790444 ip-10-0-128-161 kubenswrapper[4810]: W1030 20:59:46.790418    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf728965a_371e_4587_96cf_91d70bf0d9e9.slice/crio-b4a9262be39da34d752ebfecd1afa8e8d2e27ac31186c8ad8cb489cc26fdb910 WatchSource:0}: Error finding container b4a9262be39da34d752ebfecd1afa8e8d2e27ac31186c8ad8cb489cc26fdb910: Status 404 returned error can't find the container with id b4a9262be39da34d752ebfecd1afa8e8d2e27ac31186c8ad8cb489cc26fdb910
Oct 30 20:59:46.947804 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:46.947762    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt" event={"ID":"f728965a-371e-4587-96cf-91d70bf0d9e9","Type":"ContainerStarted","Data":"b4a9262be39da34d752ebfecd1afa8e8d2e27ac31186c8ad8cb489cc26fdb910"}
Oct 30 20:59:47.958599 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:47.958512    4810 generic.go:356] "Generic (PLEG): container finished" podID="f728965a-371e-4587-96cf-91d70bf0d9e9" containerID="c23094d4f08cb58052d945e28a0dcaaa0ebfbfff665b970048a1932190d64226" exitCode=0
Oct 30 20:59:47.958979 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:47.958595    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt" event={"ID":"f728965a-371e-4587-96cf-91d70bf0d9e9","Type":"ContainerDied","Data":"c23094d4f08cb58052d945e28a0dcaaa0ebfbfff665b970048a1932190d64226"}
Oct 30 20:59:49.167480 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.167457    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:49.262706 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.262629    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-xks-ca\") pod \"f728965a-371e-4587-96cf-91d70bf0d9e9\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") "
Oct 30 20:59:49.262881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.262803    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f728965a-371e-4587-96cf-91d70bf0d9e9-kube-api-access-xnznw\" (UniqueName: \"kubernetes.io/projected/f728965a-371e-4587-96cf-91d70bf0d9e9-kube-api-access-xnznw\") pod \"f728965a-371e-4587-96cf-91d70bf0d9e9\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") "
Oct 30 20:59:49.262881 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.262833    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-openshift-ca\") pod \"f728965a-371e-4587-96cf-91d70bf0d9e9\" (UID: \"f728965a-371e-4587-96cf-91d70bf0d9e9\") "
Oct 30 20:59:49.263259 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.263230    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-openshift-ca" pod "f728965a-371e-4587-96cf-91d70bf0d9e9" (UID: "f728965a-371e-4587-96cf-91d70bf0d9e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:49.264958 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.264936    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-xks-ca" pod "f728965a-371e-4587-96cf-91d70bf0d9e9" (UID: "f728965a-371e-4587-96cf-91d70bf0d9e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:49.265094 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.265058    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f728965a-371e-4587-96cf-91d70bf0d9e9-kube-api-access-xnznw" pod "f728965a-371e-4587-96cf-91d70bf0d9e9" (UID: "f728965a-371e-4587-96cf-91d70bf0d9e9"). InnerVolumeSpecName "kube-api-access-xnznw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:49.365432 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.365402    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnznw\" (UniqueName: \"kubernetes.io/projected/f728965a-371e-4587-96cf-91d70bf0d9e9-kube-api-access-xnznw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:49.365432 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.365427    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:49.365432 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.365437    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f728965a-371e-4587-96cf-91d70bf0d9e9-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:59:49.981635 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.981548    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"
Oct 30 20:59:49.981635 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.981563    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt" event={"ID":"f728965a-371e-4587-96cf-91d70bf0d9e9","Type":"ContainerDied","Data":"b4a9262be39da34d752ebfecd1afa8e8d2e27ac31186c8ad8cb489cc26fdb910"}
Oct 30 20:59:49.981635 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:49.981618    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4a9262be39da34d752ebfecd1afa8e8d2e27ac31186c8ad8cb489cc26fdb910"
Oct 30 20:59:51.880512 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:51.879901    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"]
Oct 30 20:59:51.881032 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:51.880538    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-70cf03-v2965"]
Oct 30 20:59:53.180565 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:53.180527    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"]
Oct 30 20:59:53.184721 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:53.184691    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99b00f-bqwgc"]
Oct 30 20:59:53.672656 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:53.672619    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e4ca75-63ad-4823-a138-69e27c6403f0" path="/var/lib/kubelet/pods/c5e4ca75-63ad-4823-a138-69e27c6403f0/volumes"
Oct 30 20:59:53.673169 ip-10-0-128-161 kubenswrapper[4810]: I1030 20:59:53.673152    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb887446-5bc9-4fd7-b6a8-5052ad159859" path="/var/lib/kubelet/pods/eb887446-5bc9-4fd7-b6a8-5052ad159859/volumes"
Oct 30 20:59:55.158328 ip-10-0-128-161 kubenswrapper[4810]: E1030 20:59:55.158288    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:00:00.144220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.144190    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"]
Oct 30 21:00:00.145060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.145040    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f728965a-371e-4587-96cf-91d70bf0d9e9" containerName="revoke-keys"
Oct 30 21:00:00.145190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.145094    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f728965a-371e-4587-96cf-91d70bf0d9e9" containerName="revoke-keys"
Oct 30 21:00:00.145315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.145301    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f728965a-371e-4587-96cf-91d70bf0d9e9" containerName="revoke-keys"
Oct 30 21:00:00.150269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.150249    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"]
Oct 30 21:00:00.150394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.150372    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"
Oct 30 21:00:00.154524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.154506    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"]
Oct 30 21:00:00.154624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.154608    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"
Oct 30 21:00:00.158391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.158375    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"
Oct 30 21:00:00.173259 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.172626    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"]
Oct 30 21:00:00.178953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.178930    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"]
Oct 30 21:00:00.183285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.183229    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"]
Oct 30 21:00:00.202048 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.193489    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"]
Oct 30 21:00:00.205385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.205360    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"
Oct 30 21:00:00.220878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.219080    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"]
Oct 30 21:00:00.222935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.222909    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"]
Oct 30 21:00:00.227980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.227325    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b647bc8-a566-44b6-962d-a6703b77e32d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr\" (UID: \"3b647bc8-a566-44b6-962d-a6703b77e32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"
Oct 30 21:00:00.227980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.227371    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsg5\" (UniqueName: \"kubernetes.io/projected/3b647bc8-a566-44b6-962d-a6703b77e32d-kube-api-access-vdsg5\") pod \"maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr\" (UID: \"3b647bc8-a566-44b6-962d-a6703b77e32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"
Oct 30 21:00:00.227980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.227406    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/749912b0-f293-47d3-a6e8-a38df024c415-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s\" (UID: \"749912b0-f293-47d3-a6e8-a38df024c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"
Oct 30 21:00:00.227980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.227534    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87njw\" (UniqueName: \"kubernetes.io/projected/749912b0-f293-47d3-a6e8-a38df024c415-kube-api-access-87njw\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s\" (UID: \"749912b0-f293-47d3-a6e8-a38df024c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"
Oct 30 21:00:00.227980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.227632    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4\" (UID: \"6ae018d7-4f7d-4fb6-a441-f3f9785f2873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"
Oct 30 21:00:00.227980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.227700    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxxtg\" (UniqueName: \"kubernetes.io/projected/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-kube-api-access-hxxtg\") pod \"maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4\" (UID: \"6ae018d7-4f7d-4fb6-a441-f3f9785f2873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"
Oct 30 21:00:00.229549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.229114    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"
Oct 30 21:00:00.236045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.236029    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"]
Oct 30 21:00:00.241173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.241153    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:00:00.328745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.328715    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pclx\" (UniqueName: \"kubernetes.io/projected/1c7dff60-eef4-42ac-863d-7f8de225ead2-kube-api-access-4pclx\") pod \"maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l\" (UID: \"1c7dff60-eef4-42ac-863d-7f8de225ead2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"
Oct 30 21:00:00.328944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.328756    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87njw\" (UniqueName: \"kubernetes.io/projected/749912b0-f293-47d3-a6e8-a38df024c415-kube-api-access-87njw\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s\" (UID: \"749912b0-f293-47d3-a6e8-a38df024c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"
Oct 30 21:00:00.328944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.328819    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e04e58d-8341-41ec-8270-b670c008fbee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc\" (UID: \"4e04e58d-8341-41ec-8270-b670c008fbee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"
Oct 30 21:00:00.328944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.328901    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbk8l\" (UniqueName: \"kubernetes.io/projected/4e04e58d-8341-41ec-8270-b670c008fbee-kube-api-access-wbk8l\") pod \"maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc\" (UID: \"4e04e58d-8341-41ec-8270-b670c008fbee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"
Oct 30 21:00:00.329070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.328989    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4\" (UID: \"6ae018d7-4f7d-4fb6-a441-f3f9785f2873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"
Oct 30 21:00:00.329070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.329031    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7dff60-eef4-42ac-863d-7f8de225ead2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l\" (UID: \"1c7dff60-eef4-42ac-863d-7f8de225ead2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"
Oct 30 21:00:00.329138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.329091    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxxtg\" (UniqueName: \"kubernetes.io/projected/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-kube-api-access-hxxtg\") pod \"maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4\" (UID: \"6ae018d7-4f7d-4fb6-a441-f3f9785f2873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"
Oct 30 21:00:00.329247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.329226    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b647bc8-a566-44b6-962d-a6703b77e32d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr\" (UID: \"3b647bc8-a566-44b6-962d-a6703b77e32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"
Oct 30 21:00:00.329305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.329261    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsg5\" (UniqueName: \"kubernetes.io/projected/3b647bc8-a566-44b6-962d-a6703b77e32d-kube-api-access-vdsg5\") pod \"maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr\" (UID: \"3b647bc8-a566-44b6-962d-a6703b77e32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"
Oct 30 21:00:00.329305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.329292    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/749912b0-f293-47d3-a6e8-a38df024c415-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s\" (UID: \"749912b0-f293-47d3-a6e8-a38df024c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"
Oct 30 21:00:00.329652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.329631    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4\" (UID: \"6ae018d7-4f7d-4fb6-a441-f3f9785f2873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"
Oct 30 21:00:00.330045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.330027    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/749912b0-f293-47d3-a6e8-a38df024c415-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s\" (UID: \"749912b0-f293-47d3-a6e8-a38df024c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"
Oct 30 21:00:00.330434 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.330419    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b647bc8-a566-44b6-962d-a6703b77e32d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr\" (UID: \"3b647bc8-a566-44b6-962d-a6703b77e32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"
Oct 30 21:00:00.337787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.337761    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87njw\" (UniqueName: \"kubernetes.io/projected/749912b0-f293-47d3-a6e8-a38df024c415-kube-api-access-87njw\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s\" (UID: \"749912b0-f293-47d3-a6e8-a38df024c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"
Oct 30 21:00:00.337914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.337904    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsg5\" (UniqueName: \"kubernetes.io/projected/3b647bc8-a566-44b6-962d-a6703b77e32d-kube-api-access-vdsg5\") pod \"maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr\" (UID: \"3b647bc8-a566-44b6-962d-a6703b77e32d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"
Oct 30 21:00:00.338016 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.337982    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxxtg\" (UniqueName: \"kubernetes.io/projected/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-kube-api-access-hxxtg\") pod \"maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4\" (UID: \"6ae018d7-4f7d-4fb6-a441-f3f9785f2873\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"
Oct 30 21:00:00.430107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.430031    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbk8l\" (UniqueName: \"kubernetes.io/projected/4e04e58d-8341-41ec-8270-b670c008fbee-kube-api-access-wbk8l\") pod \"maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc\" (UID: \"4e04e58d-8341-41ec-8270-b670c008fbee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"
Oct 30 21:00:00.430107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.430068    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7dff60-eef4-42ac-863d-7f8de225ead2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l\" (UID: \"1c7dff60-eef4-42ac-863d-7f8de225ead2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"
Oct 30 21:00:00.430306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.430182    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pclx\" (UniqueName: \"kubernetes.io/projected/1c7dff60-eef4-42ac-863d-7f8de225ead2-kube-api-access-4pclx\") pod \"maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l\" (UID: \"1c7dff60-eef4-42ac-863d-7f8de225ead2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"
Oct 30 21:00:00.430306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.430217    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e04e58d-8341-41ec-8270-b670c008fbee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc\" (UID: \"4e04e58d-8341-41ec-8270-b670c008fbee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"
Oct 30 21:00:00.430723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.430704    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7dff60-eef4-42ac-863d-7f8de225ead2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l\" (UID: \"1c7dff60-eef4-42ac-863d-7f8de225ead2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"
Oct 30 21:00:00.430807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.430791    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e04e58d-8341-41ec-8270-b670c008fbee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc\" (UID: \"4e04e58d-8341-41ec-8270-b670c008fbee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"
Oct 30 21:00:00.441128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.441084    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pclx\" (UniqueName: \"kubernetes.io/projected/1c7dff60-eef4-42ac-863d-7f8de225ead2-kube-api-access-4pclx\") pod \"maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l\" (UID: \"1c7dff60-eef4-42ac-863d-7f8de225ead2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"
Oct 30 21:00:00.441128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.441110    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbk8l\" (UniqueName: \"kubernetes.io/projected/4e04e58d-8341-41ec-8270-b670c008fbee-kube-api-access-wbk8l\") pod \"maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc\" (UID: \"4e04e58d-8341-41ec-8270-b670c008fbee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"
Oct 30 21:00:00.465605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.465578    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"
Oct 30 21:00:00.481847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.481771    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"
Oct 30 21:00:00.517167 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.515474    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"
Oct 30 21:00:00.561881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.559048    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"
Oct 30 21:00:00.567679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.567483    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"
Oct 30 21:00:00.940335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.940298    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"]
Oct 30 21:00:00.943081 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:00.943028    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749912b0_f293_47d3_a6e8_a38df024c415.slice/crio-dde477a4d199a1963adc21887f179dadc5f63c67aadbc0721b7a80e6e8864f33 WatchSource:0}: Error finding container dde477a4d199a1963adc21887f179dadc5f63c67aadbc0721b7a80e6e8864f33: Status 404 returned error can't find the container with id dde477a4d199a1963adc21887f179dadc5f63c67aadbc0721b7a80e6e8864f33
Oct 30 21:00:00.986903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:00.986541    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"]
Oct 30 21:00:01.119367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:01.119304    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s" event={"ID":"749912b0-f293-47d3-a6e8-a38df024c415","Type":"ContainerStarted","Data":"dde477a4d199a1963adc21887f179dadc5f63c67aadbc0721b7a80e6e8864f33"}
Oct 30 21:00:01.121636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:01.121329    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4" event={"ID":"6ae018d7-4f7d-4fb6-a441-f3f9785f2873","Type":"ContainerStarted","Data":"e5c1e5cea0977d62471d72855221a319a2ce4987b9175299038ca0cdd9a324dd"}
Oct 30 21:00:01.123225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:01.123193    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"]
Oct 30 21:00:01.124064 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:01.124021    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7dff60_eef4_42ac_863d_7f8de225ead2.slice/crio-c82ce945175d5f61b093cdfd1fb7d42968461288961b70f676f8f10d391000e5 WatchSource:0}: Error finding container c82ce945175d5f61b093cdfd1fb7d42968461288961b70f676f8f10d391000e5: Status 404 returned error can't find the container with id c82ce945175d5f61b093cdfd1fb7d42968461288961b70f676f8f10d391000e5
Oct 30 21:00:01.126925 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:01.126899    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e04e58d_8341_41ec_8270_b670c008fbee.slice/crio-6ab37d49f2fbda45d1c85f11b0c09e78c0e25dde713181ba21dc138bf5c9c4c3 WatchSource:0}: Error finding container 6ab37d49f2fbda45d1c85f11b0c09e78c0e25dde713181ba21dc138bf5c9c4c3: Status 404 returned error can't find the container with id 6ab37d49f2fbda45d1c85f11b0c09e78c0e25dde713181ba21dc138bf5c9c4c3
Oct 30 21:00:01.127363 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:01.127340    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"]
Oct 30 21:00:01.128215 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:01.127896    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b647bc8_a566_44b6_962d_a6703b77e32d.slice/crio-9461e471916d797614c91c2635269e887affa41c82c7cb7d33d2e1c838ca82fd WatchSource:0}: Error finding container 9461e471916d797614c91c2635269e887affa41c82c7cb7d33d2e1c838ca82fd: Status 404 returned error can't find the container with id 9461e471916d797614c91c2635269e887affa41c82c7cb7d33d2e1c838ca82fd
Oct 30 21:00:01.130670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:01.130654    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"]
Oct 30 21:00:02.221805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.221759    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l" event={"ID":"1c7dff60-eef4-42ac-863d-7f8de225ead2","Type":"ContainerStarted","Data":"41530e1f36ab6a3c20b2b2926caf3f3501f320aa2f9933479bb67f549fbb420a"}
Oct 30 21:00:02.223667 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.223639    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l" event={"ID":"1c7dff60-eef4-42ac-863d-7f8de225ead2","Type":"ContainerStarted","Data":"c82ce945175d5f61b093cdfd1fb7d42968461288961b70f676f8f10d391000e5"}
Oct 30 21:00:02.233636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.232075    4810 generic.go:356] "Generic (PLEG): container finished" podID="749912b0-f293-47d3-a6e8-a38df024c415" containerID="fa71d5af94fea0c4b2682c63fe5704a16593349f8436f053bda6ed9b4d397eb6" exitCode=60
Oct 30 21:00:02.233636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.232227    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s" event={"ID":"749912b0-f293-47d3-a6e8-a38df024c415","Type":"ContainerDied","Data":"fa71d5af94fea0c4b2682c63fe5704a16593349f8436f053bda6ed9b4d397eb6"}
Oct 30 21:00:02.233636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.232689    4810 scope.go:122] "RemoveContainer" containerID="fa71d5af94fea0c4b2682c63fe5704a16593349f8436f053bda6ed9b4d397eb6"
Oct 30 21:00:02.244879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.243941    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l" podStartSLOduration=2.243929072 podStartE2EDuration="2.243929072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.243185322 +0000 UTC m=+1368.735001891" watchObservedRunningTime="2026-10-30 21:00:02.243929072 +0000 UTC m=+1368.735745616"
Oct 30 21:00:02.273892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.270170    4810 generic.go:356] "Generic (PLEG): container finished" podID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerID="565a5b8948154d0649034250f5f537556c207f7813d65dcc4d5bd5664996f325" exitCode=60
Oct 30 21:00:02.273892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.270378    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4" event={"ID":"6ae018d7-4f7d-4fb6-a441-f3f9785f2873","Type":"ContainerDied","Data":"565a5b8948154d0649034250f5f537556c207f7813d65dcc4d5bd5664996f325"}
Oct 30 21:00:02.273892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.270797    4810 scope.go:122] "RemoveContainer" containerID="565a5b8948154d0649034250f5f537556c207f7813d65dcc4d5bd5664996f325"
Oct 30 21:00:02.285348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.285065    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr" event={"ID":"3b647bc8-a566-44b6-962d-a6703b77e32d","Type":"ContainerStarted","Data":"2e89031b7f3f9a3d75097b5b6187205e5b4ae7a747cb5d98b8834187d5a0b562"}
Oct 30 21:00:02.285348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.285122    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr" event={"ID":"3b647bc8-a566-44b6-962d-a6703b77e32d","Type":"ContainerStarted","Data":"9461e471916d797614c91c2635269e887affa41c82c7cb7d33d2e1c838ca82fd"}
Oct 30 21:00:02.317888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.317227    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" event={"ID":"4e04e58d-8341-41ec-8270-b670c008fbee","Type":"ContainerStarted","Data":"545934ded2aaff95a649598e6374fd1cc402378f137ae184db2772c10f18c186"}
Oct 30 21:00:02.317888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.317291    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" event={"ID":"4e04e58d-8341-41ec-8270-b670c008fbee","Type":"ContainerStarted","Data":"6ab37d49f2fbda45d1c85f11b0c09e78c0e25dde713181ba21dc138bf5c9c4c3"}
Oct 30 21:00:02.349394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.349337    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" podStartSLOduration=2.349316231 podStartE2EDuration="2.349316231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.346314243 +0000 UTC m=+1368.838130788" watchObservedRunningTime="2026-10-30 21:00:02.349316231 +0000 UTC m=+1368.841132776"
Oct 30 21:00:02.368302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:02.368235    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr" podStartSLOduration=2.3682143460000002 podStartE2EDuration="2.368214346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.362701627 +0000 UTC m=+1368.854518166" watchObservedRunningTime="2026-10-30 21:00:02.368214346 +0000 UTC m=+1368.860030892"
Oct 30 21:00:02.388567 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:02.388499    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae018d7_4f7d_4fb6_a441_f3f9785f2873.slice/crio-565a5b8948154d0649034250f5f537556c207f7813d65dcc4d5bd5664996f325.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae018d7_4f7d_4fb6_a441_f3f9785f2873.slice/crio-conmon-565a5b8948154d0649034250f5f537556c207f7813d65dcc4d5bd5664996f325.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e04e58d_8341_41ec_8270_b670c008fbee.slice/crio-conmon-545934ded2aaff95a649598e6374fd1cc402378f137ae184db2772c10f18c186.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e04e58d_8341_41ec_8270_b670c008fbee.slice/crio-545934ded2aaff95a649598e6374fd1cc402378f137ae184db2772c10f18c186.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:00:03.326302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.326263    4810 generic.go:356] "Generic (PLEG): container finished" podID="3b647bc8-a566-44b6-962d-a6703b77e32d" containerID="2e89031b7f3f9a3d75097b5b6187205e5b4ae7a747cb5d98b8834187d5a0b562" exitCode=60
Oct 30 21:00:03.326757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.326383    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr" event={"ID":"3b647bc8-a566-44b6-962d-a6703b77e32d","Type":"ContainerDied","Data":"2e89031b7f3f9a3d75097b5b6187205e5b4ae7a747cb5d98b8834187d5a0b562"}
Oct 30 21:00:03.326757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.326644    4810 scope.go:122] "RemoveContainer" containerID="2e89031b7f3f9a3d75097b5b6187205e5b4ae7a747cb5d98b8834187d5a0b562"
Oct 30 21:00:03.328271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.328249    4810 generic.go:356] "Generic (PLEG): container finished" podID="4e04e58d-8341-41ec-8270-b670c008fbee" containerID="545934ded2aaff95a649598e6374fd1cc402378f137ae184db2772c10f18c186" exitCode=60
Oct 30 21:00:03.328394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.328274    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" event={"ID":"4e04e58d-8341-41ec-8270-b670c008fbee","Type":"ContainerDied","Data":"545934ded2aaff95a649598e6374fd1cc402378f137ae184db2772c10f18c186"}
Oct 30 21:00:03.328558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.328548    4810 scope.go:122] "RemoveContainer" containerID="545934ded2aaff95a649598e6374fd1cc402378f137ae184db2772c10f18c186"
Oct 30 21:00:03.330305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.330146    4810 generic.go:356] "Generic (PLEG): container finished" podID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerID="41530e1f36ab6a3c20b2b2926caf3f3501f320aa2f9933479bb67f549fbb420a" exitCode=60
Oct 30 21:00:03.330305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.330232    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l" event={"ID":"1c7dff60-eef4-42ac-863d-7f8de225ead2","Type":"ContainerDied","Data":"41530e1f36ab6a3c20b2b2926caf3f3501f320aa2f9933479bb67f549fbb420a"}
Oct 30 21:00:03.330481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.330444    4810 scope.go:122] "RemoveContainer" containerID="41530e1f36ab6a3c20b2b2926caf3f3501f320aa2f9933479bb67f549fbb420a"
Oct 30 21:00:03.332361 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.332336    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s" event={"ID":"749912b0-f293-47d3-a6e8-a38df024c415","Type":"ContainerStarted","Data":"fc599b939c4ba57a9545f24f4dd5e7c88793d90c66c2cfbb3d9748918b5b506d"}
Oct 30 21:00:03.334198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.334177    4810 generic.go:356] "Generic (PLEG): container finished" podID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerID="a1be18e501ca94ac768adfd33d00e358a2e3ee64bda6d94494a0ef7844116e6e" exitCode=60
Oct 30 21:00:03.334303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.334226    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4" event={"ID":"6ae018d7-4f7d-4fb6-a441-f3f9785f2873","Type":"ContainerDied","Data":"a1be18e501ca94ac768adfd33d00e358a2e3ee64bda6d94494a0ef7844116e6e"}
Oct 30 21:00:03.334303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.334266    4810 scope.go:122] "RemoveContainer" containerID="565a5b8948154d0649034250f5f537556c207f7813d65dcc4d5bd5664996f325"
Oct 30 21:00:03.334441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.334426    4810 scope.go:122] "RemoveContainer" containerID="a1be18e501ca94ac768adfd33d00e358a2e3ee64bda6d94494a0ef7844116e6e"
Oct 30 21:00:03.334669 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:03.334646    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4_odh-ai-gateway-infra(6ae018d7-4f7d-4fb6-a441-f3f9785f2873)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873"
Oct 30 21:00:03.455517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:03.446694    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s" podStartSLOduration=3.446675774 podStartE2EDuration="3.446675774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:03.443770161 +0000 UTC m=+1369.935587083" watchObservedRunningTime="2026-10-30 21:00:03.446675774 +0000 UTC m=+1369.938492319"
Oct 30 21:00:04.356912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.348801    4810 scope.go:122] "RemoveContainer" containerID="a1be18e501ca94ac768adfd33d00e358a2e3ee64bda6d94494a0ef7844116e6e"
Oct 30 21:00:04.356912 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:04.350337    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4_odh-ai-gateway-infra(6ae018d7-4f7d-4fb6-a441-f3f9785f2873)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873"
Oct 30 21:00:04.356912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.350937    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr" event={"ID":"3b647bc8-a566-44b6-962d-a6703b77e32d","Type":"ContainerStarted","Data":"6d04fadb2b210a21454a742834c9be50f7ed7cad63cf48b646c5884600b688d0"}
Oct 30 21:00:04.356912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.353635    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" event={"ID":"4e04e58d-8341-41ec-8270-b670c008fbee","Type":"ContainerStarted","Data":"a1d7c4d77b63749d5269ba6b5b19e8ecbdfea7d3f09a9525b36fff59c9ec6d4c"}
Oct 30 21:00:04.368243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.368207    4810 generic.go:356] "Generic (PLEG): container finished" podID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerID="971c1d01eeb2a3956551a40883f8cad000f04005a80d03d71b66b613548e3d9a" exitCode=60
Oct 30 21:00:04.368716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.368687    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l" event={"ID":"1c7dff60-eef4-42ac-863d-7f8de225ead2","Type":"ContainerDied","Data":"971c1d01eeb2a3956551a40883f8cad000f04005a80d03d71b66b613548e3d9a"}
Oct 30 21:00:04.368932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.368904    4810 scope.go:122] "RemoveContainer" containerID="41530e1f36ab6a3c20b2b2926caf3f3501f320aa2f9933479bb67f549fbb420a"
Oct 30 21:00:04.369188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.369170    4810 scope.go:122] "RemoveContainer" containerID="971c1d01eeb2a3956551a40883f8cad000f04005a80d03d71b66b613548e3d9a"
Oct 30 21:00:04.369482 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:04.369452    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l_odh-ai-gateway-infra(1c7dff60-eef4-42ac-863d-7f8de225ead2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2"
Oct 30 21:00:04.372676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.372654    4810 generic.go:356] "Generic (PLEG): container finished" podID="749912b0-f293-47d3-a6e8-a38df024c415" containerID="fc599b939c4ba57a9545f24f4dd5e7c88793d90c66c2cfbb3d9748918b5b506d" exitCode=60
Oct 30 21:00:04.372787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.372688    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s" event={"ID":"749912b0-f293-47d3-a6e8-a38df024c415","Type":"ContainerDied","Data":"fc599b939c4ba57a9545f24f4dd5e7c88793d90c66c2cfbb3d9748918b5b506d"}
Oct 30 21:00:04.373075 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.373059    4810 scope.go:122] "RemoveContainer" containerID="fc599b939c4ba57a9545f24f4dd5e7c88793d90c66c2cfbb3d9748918b5b506d"
Oct 30 21:00:04.373297 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:04.373276    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s_odh-ai-gateway-infra(749912b0-f293-47d3-a6e8-a38df024c415)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s" podUID="749912b0-f293-47d3-a6e8-a38df024c415"
Oct 30 21:00:04.436220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:04.436195    4810 scope.go:122] "RemoveContainer" containerID="fa71d5af94fea0c4b2682c63fe5704a16593349f8436f053bda6ed9b4d397eb6"
Oct 30 21:00:05.212014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.211622    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"]
Oct 30 21:00:05.445880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.445052    4810 scope.go:122] "RemoveContainer" containerID="fc599b939c4ba57a9545f24f4dd5e7c88793d90c66c2cfbb3d9748918b5b506d"
Oct 30 21:00:05.445880 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:05.445419    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s_odh-ai-gateway-infra(749912b0-f293-47d3-a6e8-a38df024c415)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s" podUID="749912b0-f293-47d3-a6e8-a38df024c415"
Oct 30 21:00:05.466535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.464265    4810 generic.go:356] "Generic (PLEG): container finished" podID="3b647bc8-a566-44b6-962d-a6703b77e32d" containerID="6d04fadb2b210a21454a742834c9be50f7ed7cad63cf48b646c5884600b688d0" exitCode=60
Oct 30 21:00:05.466535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.464449    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr" event={"ID":"3b647bc8-a566-44b6-962d-a6703b77e32d","Type":"ContainerDied","Data":"6d04fadb2b210a21454a742834c9be50f7ed7cad63cf48b646c5884600b688d0"}
Oct 30 21:00:05.466535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.464529    4810 scope.go:122] "RemoveContainer" containerID="2e89031b7f3f9a3d75097b5b6187205e5b4ae7a747cb5d98b8834187d5a0b562"
Oct 30 21:00:05.472879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.471976    4810 scope.go:122] "RemoveContainer" containerID="6d04fadb2b210a21454a742834c9be50f7ed7cad63cf48b646c5884600b688d0"
Oct 30 21:00:05.472879 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:05.472398    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr_odh-ai-gateway-infra(3b647bc8-a566-44b6-962d-a6703b77e32d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr" podUID="3b647bc8-a566-44b6-962d-a6703b77e32d"
Oct 30 21:00:05.550918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.546685    4810 generic.go:356] "Generic (PLEG): container finished" podID="4e04e58d-8341-41ec-8270-b670c008fbee" containerID="a1d7c4d77b63749d5269ba6b5b19e8ecbdfea7d3f09a9525b36fff59c9ec6d4c" exitCode=60
Oct 30 21:00:05.550918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.546911    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" event={"ID":"4e04e58d-8341-41ec-8270-b670c008fbee","Type":"ContainerDied","Data":"a1d7c4d77b63749d5269ba6b5b19e8ecbdfea7d3f09a9525b36fff59c9ec6d4c"}
Oct 30 21:00:05.550918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.547478    4810 scope.go:122] "RemoveContainer" containerID="a1d7c4d77b63749d5269ba6b5b19e8ecbdfea7d3f09a9525b36fff59c9ec6d4c"
Oct 30 21:00:05.550918 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:05.547947    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc_odh-ai-gateway-infra(4e04e58d-8341-41ec-8270-b670c008fbee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" podUID="4e04e58d-8341-41ec-8270-b670c008fbee"
Oct 30 21:00:05.575879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.567759    4810 scope.go:122] "RemoveContainer" containerID="971c1d01eeb2a3956551a40883f8cad000f04005a80d03d71b66b613548e3d9a"
Oct 30 21:00:05.575879 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:05.568144    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l_odh-ai-gateway-infra(1c7dff60-eef4-42ac-863d-7f8de225ead2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2"
Oct 30 21:00:05.587100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.583237    4810 scope.go:122] "RemoveContainer" containerID="545934ded2aaff95a649598e6374fd1cc402378f137ae184db2772c10f18c186"
Oct 30 21:00:05.592997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.591022    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"]
Oct 30 21:00:05.593165 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.593032    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6c1acd-bsmjt"]
Oct 30 21:00:05.663440 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.663409    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f728965a-371e-4587-96cf-91d70bf0d9e9" path="/var/lib/kubelet/pods/f728965a-371e-4587-96cf-91d70bf0d9e9/volumes"
Oct 30 21:00:05.819080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.819005    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"]
Oct 30 21:00:05.825291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.825265    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:05.830061 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.830032    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ad8d75b-c2j69\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:05.830212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.830174    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zk7v\" (UniqueName: \"kubernetes.io/projected/f8ba0110-f784-4317-abe4-fb0d7545bf25-kube-api-access-4zk7v\") pod \"maas-api-revoke-sub-6ad8d75b-c2j69\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:05.830385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.830364    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ad8d75b-c2j69\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:05.831059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.831036    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:05.834638 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.834619    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"]
Oct 30 21:00:05.931173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.931133    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zk7v\" (UniqueName: \"kubernetes.io/projected/f8ba0110-f784-4317-abe4-fb0d7545bf25-kube-api-access-4zk7v\") pod \"maas-api-revoke-sub-6ad8d75b-c2j69\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:05.931376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.931242    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ad8d75b-c2j69\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:05.931376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.931337    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ad8d75b-c2j69\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:05.932046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.932024    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ad8d75b-c2j69\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:05.933695 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.933678    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ad8d75b-c2j69\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:05.944365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:05.944336    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zk7v\" (UniqueName: \"kubernetes.io/projected/f8ba0110-f784-4317-abe4-fb0d7545bf25-kube-api-access-4zk7v\") pod \"maas-api-revoke-sub-6ad8d75b-c2j69\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:06.145019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:06.144542    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:06.435742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:06.435694    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"]
Oct 30 21:00:06.436013 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:06.435984    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ba0110_f784_4317_abe4_fb0d7545bf25.slice/crio-5a530fa9114604364d5a7378bf498d3c38eb4ca8813024c17a671e2145bda1a8 WatchSource:0}: Error finding container 5a530fa9114604364d5a7378bf498d3c38eb4ca8813024c17a671e2145bda1a8: Status 404 returned error can't find the container with id 5a530fa9114604364d5a7378bf498d3c38eb4ca8813024c17a671e2145bda1a8
Oct 30 21:00:06.575828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:06.575798    4810 scope.go:122] "RemoveContainer" containerID="a1d7c4d77b63749d5269ba6b5b19e8ecbdfea7d3f09a9525b36fff59c9ec6d4c"
Oct 30 21:00:06.576247 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:06.576143    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc_odh-ai-gateway-infra(4e04e58d-8341-41ec-8270-b670c008fbee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" podUID="4e04e58d-8341-41ec-8270-b670c008fbee"
Oct 30 21:00:06.576928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:06.576892    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69" event={"ID":"f8ba0110-f784-4317-abe4-fb0d7545bf25","Type":"ContainerStarted","Data":"5a530fa9114604364d5a7378bf498d3c38eb4ca8813024c17a671e2145bda1a8"}
Oct 30 21:00:06.724231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:06.724203    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"
Oct 30 21:00:06.912050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:06.912016    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b647bc8-a566-44b6-962d-a6703b77e32d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b647bc8-a566-44b6-962d-a6703b77e32d-maas-api-service-ca\") pod \"3b647bc8-a566-44b6-962d-a6703b77e32d\" (UID: \"3b647bc8-a566-44b6-962d-a6703b77e32d\") "
Oct 30 21:00:06.912254 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:06.912063    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b647bc8-a566-44b6-962d-a6703b77e32d-kube-api-access-vdsg5\" (UniqueName: \"kubernetes.io/projected/3b647bc8-a566-44b6-962d-a6703b77e32d-kube-api-access-vdsg5\") pod \"3b647bc8-a566-44b6-962d-a6703b77e32d\" (UID: \"3b647bc8-a566-44b6-962d-a6703b77e32d\") "
Oct 30 21:00:06.914132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:06.914098    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b647bc8-a566-44b6-962d-a6703b77e32d-maas-api-service-ca" pod "3b647bc8-a566-44b6-962d-a6703b77e32d" (UID: "3b647bc8-a566-44b6-962d-a6703b77e32d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:06.976945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:06.976889    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b647bc8-a566-44b6-962d-a6703b77e32d-kube-api-access-vdsg5" pod "3b647bc8-a566-44b6-962d-a6703b77e32d" (UID: "3b647bc8-a566-44b6-962d-a6703b77e32d"). InnerVolumeSpecName "kube-api-access-vdsg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:07.013707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:07.013672    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b647bc8-a566-44b6-962d-a6703b77e32d-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:07.013916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:07.013896    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdsg5\" (UniqueName: \"kubernetes.io/projected/3b647bc8-a566-44b6-962d-a6703b77e32d-kube-api-access-vdsg5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:07.591011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:07.590984    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"
Oct 30 21:00:07.591393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:07.591093    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr" event={"ID":"3b647bc8-a566-44b6-962d-a6703b77e32d","Type":"ContainerDied","Data":"9461e471916d797614c91c2635269e887affa41c82c7cb7d33d2e1c838ca82fd"}
Oct 30 21:00:07.591393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:07.591155    4810 scope.go:122] "RemoveContainer" containerID="6d04fadb2b210a21454a742834c9be50f7ed7cad63cf48b646c5884600b688d0"
Oct 30 21:00:07.593613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:07.593568    4810 generic.go:356] "Generic (PLEG): container finished" podID="f8ba0110-f784-4317-abe4-fb0d7545bf25" containerID="652d846199aed87fccabece38986e166f7ed1d525931851692c2ba60de50cb4f" exitCode=0
Oct 30 21:00:07.593748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:07.593730    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69" event={"ID":"f8ba0110-f784-4317-abe4-fb0d7545bf25","Type":"ContainerDied","Data":"652d846199aed87fccabece38986e166f7ed1d525931851692c2ba60de50cb4f"}
Oct 30 21:00:07.669491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:07.669456    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"]
Oct 30 21:00:07.669684 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:07.669506    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6c1acd-29889900-mnpwr"]
Oct 30 21:00:08.766999 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:08.766971    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:08.843312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:08.843280    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-xks-ca\") pod \"f8ba0110-f784-4317-abe4-fb0d7545bf25\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") "
Oct 30 21:00:08.843573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:08.843450    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ba0110-f784-4317-abe4-fb0d7545bf25-kube-api-access-4zk7v\" (UniqueName: \"kubernetes.io/projected/f8ba0110-f784-4317-abe4-fb0d7545bf25-kube-api-access-4zk7v\") pod \"f8ba0110-f784-4317-abe4-fb0d7545bf25\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") "
Oct 30 21:00:08.843573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:08.843497    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-openshift-ca\") pod \"f8ba0110-f784-4317-abe4-fb0d7545bf25\" (UID: \"f8ba0110-f784-4317-abe4-fb0d7545bf25\") "
Oct 30 21:00:08.844017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:08.843985    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-openshift-ca" pod "f8ba0110-f784-4317-abe4-fb0d7545bf25" (UID: "f8ba0110-f784-4317-abe4-fb0d7545bf25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:08.844174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:08.844153    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:08.845757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:08.845733    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-xks-ca" pod "f8ba0110-f784-4317-abe4-fb0d7545bf25" (UID: "f8ba0110-f784-4317-abe4-fb0d7545bf25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:08.849730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:08.849705    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ba0110-f784-4317-abe4-fb0d7545bf25-kube-api-access-4zk7v" pod "f8ba0110-f784-4317-abe4-fb0d7545bf25" (UID: "f8ba0110-f784-4317-abe4-fb0d7545bf25"). InnerVolumeSpecName "kube-api-access-4zk7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:08.945366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:08.945332    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zk7v\" (UniqueName: \"kubernetes.io/projected/f8ba0110-f784-4317-abe4-fb0d7545bf25-kube-api-access-4zk7v\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:08.945366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:08.945361    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ba0110-f784-4317-abe4-fb0d7545bf25-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:09.611339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:09.611311    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"
Oct 30 21:00:09.611513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:09.611333    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69" event={"ID":"f8ba0110-f784-4317-abe4-fb0d7545bf25","Type":"ContainerDied","Data":"5a530fa9114604364d5a7378bf498d3c38eb4ca8813024c17a671e2145bda1a8"}
Oct 30 21:00:09.611513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:09.611382    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a530fa9114604364d5a7378bf498d3c38eb4ca8813024c17a671e2145bda1a8"
Oct 30 21:00:09.670609 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:09.670578    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b647bc8-a566-44b6-962d-a6703b77e32d" path="/var/lib/kubelet/pods/3b647bc8-a566-44b6-962d-a6703b77e32d/volumes"
Oct 30 21:00:11.024547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:11.024499    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"]
Oct 30 21:00:11.027390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:11.027355    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322586cd-nlw89"]
Oct 30 21:00:11.662459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:11.662428    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1124a4d8-de7b-45f0-8ffc-fd7974ea2117" path="/var/lib/kubelet/pods/1124a4d8-de7b-45f0-8ffc-fd7974ea2117/volumes"
Oct 30 21:00:14.986712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:14.986669    4810 scope.go:122] "RemoveContainer" containerID="20e1b305c36dbc2c00f5243307cf14d988c928676541e1446b9d79944194316c"
Oct 30 21:00:14.997641 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:14.997618    4810 scope.go:122] "RemoveContainer" containerID="03bf8e05a5b15ad3b1e3297210550b96f4d2726a1063508c94b84f7631594ec2"
Oct 30 21:00:15.008218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.008199    4810 scope.go:122] "RemoveContainer" containerID="03977cd79ec23daea09ef44cc13845b58b7e838b70cd2038c6d4cd5cbe197501"
Oct 30 21:00:15.018696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.018471    4810 scope.go:122] "RemoveContainer" containerID="5d9a2d5a82e52d4d7fc54015b74fbece530ea392609ead486476232c050962f5"
Oct 30 21:00:15.058834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.058686    4810 scope.go:122] "RemoveContainer" containerID="95037646660c01003b288eff4d5972d583cc59429a2d259c9497cd6a32d1d3a4"
Oct 30 21:00:15.260338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.260251    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"]
Oct 30 21:00:15.261026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.261006    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3b647bc8-a566-44b6-962d-a6703b77e32d" containerName="cleanup"
Oct 30 21:00:15.261026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.261026    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b647bc8-a566-44b6-962d-a6703b77e32d" containerName="cleanup"
Oct 30 21:00:15.261208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.261041    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f8ba0110-f784-4317-abe4-fb0d7545bf25" containerName="revoke-keys"
Oct 30 21:00:15.261208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.261049    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ba0110-f784-4317-abe4-fb0d7545bf25" containerName="revoke-keys"
Oct 30 21:00:15.261208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.261065    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3b647bc8-a566-44b6-962d-a6703b77e32d" containerName="cleanup"
Oct 30 21:00:15.261208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.261072    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b647bc8-a566-44b6-962d-a6703b77e32d" containerName="cleanup"
Oct 30 21:00:15.261414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.261220    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b647bc8-a566-44b6-962d-a6703b77e32d" containerName="cleanup"
Oct 30 21:00:15.261414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.261236    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ba0110-f784-4317-abe4-fb0d7545bf25" containerName="revoke-keys"
Oct 30 21:00:15.261414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.261247    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b647bc8-a566-44b6-962d-a6703b77e32d" containerName="cleanup"
Oct 30 21:00:15.266902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.266871    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.271430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.271408    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:15.282922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.282838    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"]
Oct 30 21:00:15.351161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.351129    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e40f1db9-k28bz\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.351319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.351270    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e40f1db9-k28bz\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.351319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.351308    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55pg\" (UniqueName: \"kubernetes.io/projected/11777510-934a-4d08-96bf-3e00a8ae19c7-kube-api-access-t55pg\") pod \"maas-api-revoke-sub-e40f1db9-k28bz\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.452437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.452401    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e40f1db9-k28bz\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.452611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.452506    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e40f1db9-k28bz\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.452611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.452536    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t55pg\" (UniqueName: \"kubernetes.io/projected/11777510-934a-4d08-96bf-3e00a8ae19c7-kube-api-access-t55pg\") pod \"maas-api-revoke-sub-e40f1db9-k28bz\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.453116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.453089    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e40f1db9-k28bz\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.454825 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.454807    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e40f1db9-k28bz\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.460912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.460892    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55pg\" (UniqueName: \"kubernetes.io/projected/11777510-934a-4d08-96bf-3e00a8ae19c7-kube-api-access-t55pg\") pod \"maas-api-revoke-sub-e40f1db9-k28bz\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.587416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.587313    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:15.675868 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.675804    4810 scope.go:122] "RemoveContainer" containerID="a1d7c4d77b63749d5269ba6b5b19e8ecbdfea7d3f09a9525b36fff59c9ec6d4c"
Oct 30 21:00:15.867443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:15.867418    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"]
Oct 30 21:00:15.868179 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:15.868154    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11777510_934a_4d08_96bf_3e00a8ae19c7.slice/crio-f45c5ac8918a2d24c2c117995aa1a9818ebb2aaebda3f370a3a6139f90fad5db WatchSource:0}: Error finding container f45c5ac8918a2d24c2c117995aa1a9818ebb2aaebda3f370a3a6139f90fad5db: Status 404 returned error can't find the container with id f45c5ac8918a2d24c2c117995aa1a9818ebb2aaebda3f370a3a6139f90fad5db
Oct 30 21:00:16.654954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:16.654924    4810 scope.go:122] "RemoveContainer" containerID="fc599b939c4ba57a9545f24f4dd5e7c88793d90c66c2cfbb3d9748918b5b506d"
Oct 30 21:00:16.733981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:16.733940    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz" event={"ID":"11777510-934a-4d08-96bf-3e00a8ae19c7","Type":"ContainerStarted","Data":"04198f2bd4a7f36be78136dfb2054e2dc8274e886e03e2292a91d325ea88b6af"}
Oct 30 21:00:16.734134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:16.734003    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz" event={"ID":"11777510-934a-4d08-96bf-3e00a8ae19c7","Type":"ContainerStarted","Data":"f45c5ac8918a2d24c2c117995aa1a9818ebb2aaebda3f370a3a6139f90fad5db"}
Oct 30 21:00:16.735898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:16.735872    4810 generic.go:356] "Generic (PLEG): container finished" podID="4e04e58d-8341-41ec-8270-b670c008fbee" containerID="1083cb647557e8fd0c944aee4020fedeee413da1ccbb612d6554ef2820a259ce" exitCode=60
Oct 30 21:00:16.735898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:16.735884    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" event={"ID":"4e04e58d-8341-41ec-8270-b670c008fbee","Type":"ContainerDied","Data":"1083cb647557e8fd0c944aee4020fedeee413da1ccbb612d6554ef2820a259ce"}
Oct 30 21:00:16.736054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:16.735944    4810 scope.go:122] "RemoveContainer" containerID="a1d7c4d77b63749d5269ba6b5b19e8ecbdfea7d3f09a9525b36fff59c9ec6d4c"
Oct 30 21:00:16.736284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:16.736268    4810 scope.go:122] "RemoveContainer" containerID="1083cb647557e8fd0c944aee4020fedeee413da1ccbb612d6554ef2820a259ce"
Oct 30 21:00:16.736572 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:16.736556    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc_odh-ai-gateway-infra(4e04e58d-8341-41ec-8270-b670c008fbee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" podUID="4e04e58d-8341-41ec-8270-b670c008fbee"
Oct 30 21:00:16.753797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:16.753746    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz" podStartSLOduration=1.753729938 podStartE2EDuration="1.753729938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:16.749550328 +0000 UTC m=+1383.241366874" watchObservedRunningTime="2026-10-30 21:00:16.753729938 +0000 UTC m=+1383.245546484"
Oct 30 21:00:17.511925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.511892    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:00:17.670787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.670733    4810 scope.go:122] "RemoveContainer" containerID="a1be18e501ca94ac768adfd33d00e358a2e3ee64bda6d94494a0ef7844116e6e"
Oct 30 21:00:17.674072 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.674046    4810 scope.go:122] "RemoveContainer" containerID="971c1d01eeb2a3956551a40883f8cad000f04005a80d03d71b66b613548e3d9a"
Oct 30 21:00:17.748776 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.748752    4810 generic.go:356] "Generic (PLEG): container finished" podID="749912b0-f293-47d3-a6e8-a38df024c415" containerID="59ecee27bdf8f7f8c788a496e20beff0877c2dbd7041d84c30f4e27ad6041fb1" exitCode=60
Oct 30 21:00:17.748909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.748829    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s" event={"ID":"749912b0-f293-47d3-a6e8-a38df024c415","Type":"ContainerDied","Data":"59ecee27bdf8f7f8c788a496e20beff0877c2dbd7041d84c30f4e27ad6041fb1"}
Oct 30 21:00:17.748999 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.748962    4810 scope.go:122] "RemoveContainer" containerID="fc599b939c4ba57a9545f24f4dd5e7c88793d90c66c2cfbb3d9748918b5b506d"
Oct 30 21:00:17.749380 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.749362    4810 scope.go:122] "RemoveContainer" containerID="59ecee27bdf8f7f8c788a496e20beff0877c2dbd7041d84c30f4e27ad6041fb1"
Oct 30 21:00:17.749723 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:17.749698    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s_odh-ai-gateway-infra(749912b0-f293-47d3-a6e8-a38df024c415)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s" podUID="749912b0-f293-47d3-a6e8-a38df024c415"
Oct 30 21:00:17.750905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.750884    4810 generic.go:356] "Generic (PLEG): container finished" podID="11777510-934a-4d08-96bf-3e00a8ae19c7" containerID="04198f2bd4a7f36be78136dfb2054e2dc8274e886e03e2292a91d325ea88b6af" exitCode=0
Oct 30 21:00:17.751008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.750965    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz" event={"ID":"11777510-934a-4d08-96bf-3e00a8ae19c7","Type":"ContainerDied","Data":"04198f2bd4a7f36be78136dfb2054e2dc8274e886e03e2292a91d325ea88b6af"}
Oct 30 21:00:17.780986 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.780959    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"]
Oct 30 21:00:17.972786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:17.972756    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"
Oct 30 21:00:18.099296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.099266    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e04e58d-8341-41ec-8270-b670c008fbee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e04e58d-8341-41ec-8270-b670c008fbee-maas-api-service-ca\") pod \"4e04e58d-8341-41ec-8270-b670c008fbee\" (UID: \"4e04e58d-8341-41ec-8270-b670c008fbee\") "
Oct 30 21:00:18.099464 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.099359    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e04e58d-8341-41ec-8270-b670c008fbee-kube-api-access-wbk8l\" (UniqueName: \"kubernetes.io/projected/4e04e58d-8341-41ec-8270-b670c008fbee-kube-api-access-wbk8l\") pod \"4e04e58d-8341-41ec-8270-b670c008fbee\" (UID: \"4e04e58d-8341-41ec-8270-b670c008fbee\") "
Oct 30 21:00:18.099680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.099646    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e04e58d-8341-41ec-8270-b670c008fbee-maas-api-service-ca" pod "4e04e58d-8341-41ec-8270-b670c008fbee" (UID: "4e04e58d-8341-41ec-8270-b670c008fbee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:18.099944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.099929    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e04e58d-8341-41ec-8270-b670c008fbee-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:18.101592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.101570    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e04e58d-8341-41ec-8270-b670c008fbee-kube-api-access-wbk8l" pod "4e04e58d-8341-41ec-8270-b670c008fbee" (UID: "4e04e58d-8341-41ec-8270-b670c008fbee"). InnerVolumeSpecName "kube-api-access-wbk8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:18.203258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.201824    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbk8l\" (UniqueName: \"kubernetes.io/projected/4e04e58d-8341-41ec-8270-b670c008fbee-kube-api-access-wbk8l\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:18.795883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.783682    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"]
Oct 30 21:00:18.795883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.790735    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc" event={"ID":"4e04e58d-8341-41ec-8270-b670c008fbee","Type":"ContainerDied","Data":"6ab37d49f2fbda45d1c85f11b0c09e78c0e25dde713181ba21dc138bf5c9c4c3"}
Oct 30 21:00:18.795883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.790803    4810 scope.go:122] "RemoveContainer" containerID="1083cb647557e8fd0c944aee4020fedeee413da1ccbb612d6554ef2820a259ce"
Oct 30 21:00:18.795883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.790921    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"
Oct 30 21:00:18.795883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.794803    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l" event={"ID":"1c7dff60-eef4-42ac-863d-7f8de225ead2","Type":"ContainerStarted","Data":"b78bc98ac75f246d655d22ac1139ed8255efe14de1f762253cd1404b3253a819"}
Oct 30 21:00:18.821875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.821261    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4" event={"ID":"6ae018d7-4f7d-4fb6-a441-f3f9785f2873","Type":"ContainerStarted","Data":"62786f8c83b1508748fc656d23d761f8be348e65133092a473b4eae4300361c6"}
Oct 30 21:00:18.864176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.864122    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4" podStartSLOduration=18.864107047 podStartE2EDuration="18.864107047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:18.85720664 +0000 UTC m=+1385.349023185" watchObservedRunningTime="2026-10-30 21:00:18.864107047 +0000 UTC m=+1385.355923589"
Oct 30 21:00:18.929038 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.922679    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"]
Oct 30 21:00:18.951886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:18.950134    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b80638-29889900-zhkzc"]
Oct 30 21:00:20.202885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.188022    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e04e58d-8341-41ec-8270-b670c008fbee" path="/var/lib/kubelet/pods/4e04e58d-8341-41ec-8270-b670c008fbee/volumes"
Oct 30 21:00:20.260622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.260590    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"]
Oct 30 21:00:20.294895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.277640    4810 generic.go:356] "Generic (PLEG): container finished" podID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerID="b78bc98ac75f246d655d22ac1139ed8255efe14de1f762253cd1404b3253a819" exitCode=60
Oct 30 21:00:20.294895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.277774    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l" event={"ID":"1c7dff60-eef4-42ac-863d-7f8de225ead2","Type":"ContainerDied","Data":"b78bc98ac75f246d655d22ac1139ed8255efe14de1f762253cd1404b3253a819"}
Oct 30 21:00:20.294895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.277835    4810 scope.go:122] "RemoveContainer" containerID="971c1d01eeb2a3956551a40883f8cad000f04005a80d03d71b66b613548e3d9a"
Oct 30 21:00:20.348678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.348636    4810 generic.go:356] "Generic (PLEG): container finished" podID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerID="62786f8c83b1508748fc656d23d761f8be348e65133092a473b4eae4300361c6" exitCode=60
Oct 30 21:00:20.349068 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.348943    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4" event={"ID":"6ae018d7-4f7d-4fb6-a441-f3f9785f2873","Type":"ContainerDied","Data":"62786f8c83b1508748fc656d23d761f8be348e65133092a473b4eae4300361c6"}
Oct 30 21:00:20.350096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.350068    4810 scope.go:122] "RemoveContainer" containerID="62786f8c83b1508748fc656d23d761f8be348e65133092a473b4eae4300361c6"
Oct 30 21:00:20.350732 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:20.350689    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4_odh-ai-gateway-infra(6ae018d7-4f7d-4fb6-a441-f3f9785f2873)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873"
Oct 30 21:00:20.407883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.391833    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:20.438845 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.438807    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:20.467071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.466915    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"]
Oct 30 21:00:20.530188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.530150    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:20.609882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.564308    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11777510-934a-4d08-96bf-3e00a8ae19c7-kube-api-access-t55pg\" (UniqueName: \"kubernetes.io/projected/11777510-934a-4d08-96bf-3e00a8ae19c7-kube-api-access-t55pg\") pod \"11777510-934a-4d08-96bf-3e00a8ae19c7\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") "
Oct 30 21:00:20.632121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.632086    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-openshift-ca\") pod \"11777510-934a-4d08-96bf-3e00a8ae19c7\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") "
Oct 30 21:00:20.647126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.647089    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-xks-ca\") pod \"11777510-934a-4d08-96bf-3e00a8ae19c7\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") "
Oct 30 21:00:20.670712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.565560    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:20.670712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.650728    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-openshift-ca" pod "11777510-934a-4d08-96bf-3e00a8ae19c7" (UID: "11777510-934a-4d08-96bf-3e00a8ae19c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:20.925942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.914005    4810 scope.go:122] "RemoveContainer" containerID="a1be18e501ca94ac768adfd33d00e358a2e3ee64bda6d94494a0ef7844116e6e"
Oct 30 21:00:20.943496 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:20.943451    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-xks-ca" pod "11777510-934a-4d08-96bf-3e00a8ae19c7" (UID: "11777510-934a-4d08-96bf-3e00a8ae19c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:21.185127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.046094    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11777510-934a-4d08-96bf-3e00a8ae19c7-kube-api-access-t55pg" pod "11777510-934a-4d08-96bf-3e00a8ae19c7" (UID: "11777510-934a-4d08-96bf-3e00a8ae19c7"). InnerVolumeSpecName "kube-api-access-t55pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.185127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.112156    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:21.459946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.458873    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:21.539252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.539211    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:00:21.609876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.542972    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 21:00:21.609876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.544272    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-xks-ca\") pod \"11777510-934a-4d08-96bf-3e00a8ae19c7\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") "
Oct 30 21:00:21.610262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.610238    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 21:00:21.610419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.610405    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 21:00:21.610556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.593023    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"
Oct 30 21:00:21.610641 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:21.544340    4810 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/11777510-934a-4d08-96bf-3e00a8ae19c7/volumes/kubernetes.io~secret/maas-api-xks-ca
Oct 30 21:00:21.610641 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.610596    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-xks-ca" pod "11777510-934a-4d08-96bf-3e00a8ae19c7" (UID: "11777510-934a-4d08-96bf-3e00a8ae19c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:21.615878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.612468    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11777510-934a-4d08-96bf-3e00a8ae19c7-kube-api-access-t55pg\" (UniqueName: \"kubernetes.io/projected/11777510-934a-4d08-96bf-3e00a8ae19c7-kube-api-access-t55pg\") pod \"11777510-934a-4d08-96bf-3e00a8ae19c7\" (UID: \"11777510-934a-4d08-96bf-3e00a8ae19c7\") "
Oct 30 21:00:21.615878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.614913    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.615878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.614934    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11777510-934a-4d08-96bf-3e00a8ae19c7-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.615878 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:21.615217    4810 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/11777510-934a-4d08-96bf-3e00a8ae19c7/volumes/kubernetes.io~projected/kube-api-access-t55pg
Oct 30 21:00:21.615878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.615259    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11777510-934a-4d08-96bf-3e00a8ae19c7-kube-api-access-t55pg" pod "11777510-934a-4d08-96bf-3e00a8ae19c7" (UID: "11777510-934a-4d08-96bf-3e00a8ae19c7"). InnerVolumeSpecName "kube-api-access-t55pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.635037 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.635001    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz" event={"ID":"11777510-934a-4d08-96bf-3e00a8ae19c7","Type":"ContainerDied","Data":"f45c5ac8918a2d24c2c117995aa1a9818ebb2aaebda3f370a3a6139f90fad5db"}
Oct 30 21:00:21.635379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.635363    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f45c5ac8918a2d24c2c117995aa1a9818ebb2aaebda3f370a3a6139f90fad5db"
Oct 30 21:00:21.635472 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.635315    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"
Oct 30 21:00:21.643700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.643656    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s" event={"ID":"749912b0-f293-47d3-a6e8-a38df024c415","Type":"ContainerDied","Data":"dde477a4d199a1963adc21887f179dadc5f63c67aadbc0721b7a80e6e8864f33"}
Oct 30 21:00:21.644080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.643980    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"
Oct 30 21:00:21.644368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.644353    4810 scope.go:122] "RemoveContainer" containerID="59ecee27bdf8f7f8c788a496e20beff0877c2dbd7041d84c30f4e27ad6041fb1"
Oct 30 21:00:21.662818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.662779    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:21.715730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.715625    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749912b0-f293-47d3-a6e8-a38df024c415-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/749912b0-f293-47d3-a6e8-a38df024c415-maas-api-service-ca\") pod \"749912b0-f293-47d3-a6e8-a38df024c415\" (UID: \"749912b0-f293-47d3-a6e8-a38df024c415\") "
Oct 30 21:00:21.716169 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.716146    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749912b0-f293-47d3-a6e8-a38df024c415-kube-api-access-87njw\" (UniqueName: \"kubernetes.io/projected/749912b0-f293-47d3-a6e8-a38df024c415-kube-api-access-87njw\") pod \"749912b0-f293-47d3-a6e8-a38df024c415\" (UID: \"749912b0-f293-47d3-a6e8-a38df024c415\") "
Oct 30 21:00:21.716952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.716918    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t55pg\" (UniqueName: \"kubernetes.io/projected/11777510-934a-4d08-96bf-3e00a8ae19c7-kube-api-access-t55pg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.717789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.717765    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749912b0-f293-47d3-a6e8-a38df024c415-maas-api-service-ca" pod "749912b0-f293-47d3-a6e8-a38df024c415" (UID: "749912b0-f293-47d3-a6e8-a38df024c415"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:21.787838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.787795    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749912b0-f293-47d3-a6e8-a38df024c415-kube-api-access-87njw" pod "749912b0-f293-47d3-a6e8-a38df024c415" (UID: "749912b0-f293-47d3-a6e8-a38df024c415"). InnerVolumeSpecName "kube-api-access-87njw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.807152 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.807126    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 21:00:21.807379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.807368    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 21:00:21.807514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.807505    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 21:00:21.811760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.811450    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 21:00:21.821952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.821919    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87njw\" (UniqueName: \"kubernetes.io/projected/749912b0-f293-47d3-a6e8-a38df024c415-kube-api-access-87njw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.822106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.821955    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/749912b0-f293-47d3-a6e8-a38df024c415-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.877006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.876966    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"]
Oct 30 21:00:21.877925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.877899    4810 cpu_manager.go:414] "containerMap: removing container" podUID="749912b0-f293-47d3-a6e8-a38df024c415" containerName="cleanup"
Oct 30 21:00:21.877925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.877926    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749912b0-f293-47d3-a6e8-a38df024c415" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.877951    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4e04e58d-8341-41ec-8270-b670c008fbee" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.877959    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e04e58d-8341-41ec-8270-b670c008fbee" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.877991    4810 cpu_manager.go:414] "containerMap: removing container" podUID="749912b0-f293-47d3-a6e8-a38df024c415" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.877998    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749912b0-f293-47d3-a6e8-a38df024c415" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878035    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4e04e58d-8341-41ec-8270-b670c008fbee" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878043    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e04e58d-8341-41ec-8270-b670c008fbee" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878061    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4e04e58d-8341-41ec-8270-b670c008fbee" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878068    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e04e58d-8341-41ec-8270-b670c008fbee" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878078    4810 cpu_manager.go:414] "containerMap: removing container" podUID="11777510-934a-4d08-96bf-3e00a8ae19c7" containerName="revoke-keys"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878085    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11777510-934a-4d08-96bf-3e00a8ae19c7" containerName="revoke-keys"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878247    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="749912b0-f293-47d3-a6e8-a38df024c415" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878264    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="749912b0-f293-47d3-a6e8-a38df024c415" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878277    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e04e58d-8341-41ec-8270-b670c008fbee" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878286    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e04e58d-8341-41ec-8270-b670c008fbee" containerName="cleanup"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878300    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="11777510-934a-4d08-96bf-3e00a8ae19c7" containerName="revoke-keys"
Oct 30 21:00:21.879885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.878316    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e04e58d-8341-41ec-8270-b670c008fbee" containerName="cleanup"
Oct 30 21:00:21.884816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.884199    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"]
Oct 30 21:00:21.884816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.884310    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:21.893472 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:21.892782    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:22.051204 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.051164    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"]
Oct 30 21:00:22.053052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.052852    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:22.053414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.053396    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:22.053723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.053605    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhmwm\" (UniqueName: \"kubernetes.io/projected/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-kube-api-access-fhmwm\") pod \"maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:22.109322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.102835    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889900-qrf6s"]
Oct 30 21:00:22.114070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.114042    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"
Oct 30 21:00:22.155899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.155832    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7dff60-eef4-42ac-863d-7f8de225ead2-kube-api-access-4pclx\" (UniqueName: \"kubernetes.io/projected/1c7dff60-eef4-42ac-863d-7f8de225ead2-kube-api-access-4pclx\") pod \"1c7dff60-eef4-42ac-863d-7f8de225ead2\" (UID: \"1c7dff60-eef4-42ac-863d-7f8de225ead2\") "
Oct 30 21:00:22.156117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.155941    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7dff60-eef4-42ac-863d-7f8de225ead2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7dff60-eef4-42ac-863d-7f8de225ead2-maas-api-service-ca\") pod \"1c7dff60-eef4-42ac-863d-7f8de225ead2\" (UID: \"1c7dff60-eef4-42ac-863d-7f8de225ead2\") "
Oct 30 21:00:22.156219 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.156195    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:22.156308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.156238    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhmwm\" (UniqueName: \"kubernetes.io/projected/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-kube-api-access-fhmwm\") pod \"maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:22.156479 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.156456    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:22.157132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.157112    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:22.157425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.157406    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7dff60-eef4-42ac-863d-7f8de225ead2-maas-api-service-ca" pod "1c7dff60-eef4-42ac-863d-7f8de225ead2" (UID: "1c7dff60-eef4-42ac-863d-7f8de225ead2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:22.158096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.158079    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"
Oct 30 21:00:22.159246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.159224    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:22.160457 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.160431    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7dff60-eef4-42ac-863d-7f8de225ead2-kube-api-access-4pclx" pod "1c7dff60-eef4-42ac-863d-7f8de225ead2" (UID: "1c7dff60-eef4-42ac-863d-7f8de225ead2"). InnerVolumeSpecName "kube-api-access-4pclx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:22.164781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.164758    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhmwm\" (UniqueName: \"kubernetes.io/projected/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-kube-api-access-fhmwm\") pod \"maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:22.257304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.257268    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-maas-api-service-ca\") pod \"6ae018d7-4f7d-4fb6-a441-f3f9785f2873\" (UID: \"6ae018d7-4f7d-4fb6-a441-f3f9785f2873\") "
Oct 30 21:00:22.257478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.257321    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-kube-api-access-hxxtg\" (UniqueName: \"kubernetes.io/projected/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-kube-api-access-hxxtg\") pod \"6ae018d7-4f7d-4fb6-a441-f3f9785f2873\" (UID: \"6ae018d7-4f7d-4fb6-a441-f3f9785f2873\") "
Oct 30 21:00:22.257657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.257635    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-maas-api-service-ca" pod "6ae018d7-4f7d-4fb6-a441-f3f9785f2873" (UID: "6ae018d7-4f7d-4fb6-a441-f3f9785f2873"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:22.257836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.257822    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pclx\" (UniqueName: \"kubernetes.io/projected/1c7dff60-eef4-42ac-863d-7f8de225ead2-kube-api-access-4pclx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:22.257922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.257838    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7dff60-eef4-42ac-863d-7f8de225ead2-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:22.257922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.257847    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:22.259478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.259453    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-kube-api-access-hxxtg" pod "6ae018d7-4f7d-4fb6-a441-f3f9785f2873" (UID: "6ae018d7-4f7d-4fb6-a441-f3f9785f2873"). InnerVolumeSpecName "kube-api-access-hxxtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:22.293325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.293293    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:22.360715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.360672    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxxtg\" (UniqueName: \"kubernetes.io/projected/6ae018d7-4f7d-4fb6-a441-f3f9785f2873-kube-api-access-hxxtg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:22.612602 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:22.612555    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eac8948_5a0d_46e0_ab36_38ae2cff51bf.slice/crio-9f9f496866b58420b985856549f6e8fd6fdb92da4971ecccd8ee2a02a71b78a1 WatchSource:0}: Error finding container 9f9f496866b58420b985856549f6e8fd6fdb92da4971ecccd8ee2a02a71b78a1: Status 404 returned error can't find the container with id 9f9f496866b58420b985856549f6e8fd6fdb92da4971ecccd8ee2a02a71b78a1
Oct 30 21:00:22.613132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.613106    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"]
Oct 30 21:00:22.659683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.659649    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"
Oct 30 21:00:22.659974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.659684    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4" event={"ID":"6ae018d7-4f7d-4fb6-a441-f3f9785f2873","Type":"ContainerDied","Data":"e5c1e5cea0977d62471d72855221a319a2ce4987b9175299038ca0cdd9a324dd"}
Oct 30 21:00:22.659974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.659766    4810 scope.go:122] "RemoveContainer" containerID="62786f8c83b1508748fc656d23d761f8be348e65133092a473b4eae4300361c6"
Oct 30 21:00:22.662182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.662156    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p" event={"ID":"5eac8948-5a0d-46e0-ab36-38ae2cff51bf","Type":"ContainerStarted","Data":"9f9f496866b58420b985856549f6e8fd6fdb92da4971ecccd8ee2a02a71b78a1"}
Oct 30 21:00:22.666898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.663907    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l" event={"ID":"1c7dff60-eef4-42ac-863d-7f8de225ead2","Type":"ContainerDied","Data":"c82ce945175d5f61b093cdfd1fb7d42968461288961b70f676f8f10d391000e5"}
Oct 30 21:00:22.666898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.663936    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"
Oct 30 21:00:22.676518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.676500    4810 scope.go:122] "RemoveContainer" containerID="b78bc98ac75f246d655d22ac1139ed8255efe14de1f762253cd1404b3253a819"
Oct 30 21:00:22.685917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.685893    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"]
Oct 30 21:00:22.696639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.696608    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-955eb7-29889900-bv4w4"]
Oct 30 21:00:22.710269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.710245    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"]
Oct 30 21:00:22.716093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:22.716073    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f9a819-29889900-czt8l"]
Oct 30 21:00:23.669459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:23.669430    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2" path="/var/lib/kubelet/pods/1c7dff60-eef4-42ac-863d-7f8de225ead2/volumes"
Oct 30 21:00:23.669989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:23.669909    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" path="/var/lib/kubelet/pods/6ae018d7-4f7d-4fb6-a441-f3f9785f2873/volumes"
Oct 30 21:00:23.670575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:23.670543    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749912b0-f293-47d3-a6e8-a38df024c415" path="/var/lib/kubelet/pods/749912b0-f293-47d3-a6e8-a38df024c415/volumes"
Oct 30 21:00:23.677330 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:23.677298    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p" event={"ID":"5eac8948-5a0d-46e0-ab36-38ae2cff51bf","Type":"ContainerStarted","Data":"1991e007982b14b2ee9a4254290e9d8755ffbd8158ed22512994d31bd89f4e90"}
Oct 30 21:00:23.696190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:23.696148    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p" podStartSLOduration=2.696135149 podStartE2EDuration="2.696135149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:23.693343534 +0000 UTC m=+1390.185160078" watchObservedRunningTime="2026-10-30 21:00:23.696135149 +0000 UTC m=+1390.187951693"
Oct 30 21:00:24.689332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:24.689245    4810 generic.go:356] "Generic (PLEG): container finished" podID="5eac8948-5a0d-46e0-ab36-38ae2cff51bf" containerID="1991e007982b14b2ee9a4254290e9d8755ffbd8158ed22512994d31bd89f4e90" exitCode=0
Oct 30 21:00:24.689917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:24.689335    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p" event={"ID":"5eac8948-5a0d-46e0-ab36-38ae2cff51bf","Type":"ContainerDied","Data":"1991e007982b14b2ee9a4254290e9d8755ffbd8158ed22512994d31bd89f4e90"}
Oct 30 21:00:25.158070 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:25.158036    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:00:25.988528 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:25.988500    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:26.051961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.051922    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-xks-ca\") pod \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") "
Oct 30 21:00:26.051961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.051968    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-kube-api-access-fhmwm\" (UniqueName: \"kubernetes.io/projected/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-kube-api-access-fhmwm\") pod \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") "
Oct 30 21:00:26.052215 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.052086    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-openshift-ca\") pod \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\" (UID: \"5eac8948-5a0d-46e0-ab36-38ae2cff51bf\") "
Oct 30 21:00:26.052511 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.052473    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-openshift-ca" pod "5eac8948-5a0d-46e0-ab36-38ae2cff51bf" (UID: "5eac8948-5a0d-46e0-ab36-38ae2cff51bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:26.052773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.052744    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:26.054192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.054165    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-kube-api-access-fhmwm" pod "5eac8948-5a0d-46e0-ab36-38ae2cff51bf" (UID: "5eac8948-5a0d-46e0-ab36-38ae2cff51bf"). InnerVolumeSpecName "kube-api-access-fhmwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:26.054646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.054624    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-xks-ca" pod "5eac8948-5a0d-46e0-ab36-38ae2cff51bf" (UID: "5eac8948-5a0d-46e0-ab36-38ae2cff51bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:26.154013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.153977    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:26.154013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.154004    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhmwm\" (UniqueName: \"kubernetes.io/projected/5eac8948-5a0d-46e0-ab36-38ae2cff51bf-kube-api-access-fhmwm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:26.709168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.709141    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"
Oct 30 21:00:26.709340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.709180    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p" event={"ID":"5eac8948-5a0d-46e0-ab36-38ae2cff51bf","Type":"ContainerDied","Data":"9f9f496866b58420b985856549f6e8fd6fdb92da4971ecccd8ee2a02a71b78a1"}
Oct 30 21:00:26.709340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:26.709233    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f9f496866b58420b985856549f6e8fd6fdb92da4971ecccd8ee2a02a71b78a1"
Oct 30 21:00:27.606880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.605973    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"]
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.608877    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.608902    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.608942    4810 cpu_manager.go:414] "containerMap: removing container" podUID="749912b0-f293-47d3-a6e8-a38df024c415" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.608949    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749912b0-f293-47d3-a6e8-a38df024c415" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.608984    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.608990    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609004    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609011    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609055    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609062    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609121    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5eac8948-5a0d-46e0-ab36-38ae2cff51bf" containerName="revoke-keys"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609128    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eac8948-5a0d-46e0-ab36-38ae2cff51bf" containerName="revoke-keys"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609710    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609737    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609746    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609778    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609790    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609806    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerName="cleanup"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609821    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eac8948-5a0d-46e0-ab36-38ae2cff51bf" containerName="revoke-keys"
Oct 30 21:00:27.613910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.609839    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="749912b0-f293-47d3-a6e8-a38df024c415" containerName="cleanup"
Oct 30 21:00:27.624018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.623987    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"]
Oct 30 21:00:27.624151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.624106    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:27.628633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.628586    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:27.628849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.628824    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:00:27.775384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.775342    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e35d74-vk7nd\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:27.775384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.775399    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e35d74-vk7nd\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:27.775647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.775485    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gvb2\" (UniqueName: \"kubernetes.io/projected/ffc30df6-d7dc-437e-9bba-a1053463abea-kube-api-access-7gvb2\") pod \"maas-api-revoke-sub-11e35d74-vk7nd\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:27.876687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.876602    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e35d74-vk7nd\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:27.876687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.876660    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e35d74-vk7nd\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:27.876981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.876702    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gvb2\" (UniqueName: \"kubernetes.io/projected/ffc30df6-d7dc-437e-9bba-a1053463abea-kube-api-access-7gvb2\") pod \"maas-api-revoke-sub-11e35d74-vk7nd\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:27.877331 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.877311    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e35d74-vk7nd\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:27.879054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.879031    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e35d74-vk7nd\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:27.886504 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.886480    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gvb2\" (UniqueName: \"kubernetes.io/projected/ffc30df6-d7dc-437e-9bba-a1053463abea-kube-api-access-7gvb2\") pod \"maas-api-revoke-sub-11e35d74-vk7nd\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:27.940311 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:27.940286    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:28.224360 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.224311    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"]
Oct 30 21:00:28.225664 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:28.225637    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc30df6_d7dc_437e_9bba_a1053463abea.slice/crio-614e54ef21056f0539ceff1274cde2abe4ab832020f2409785df4ebf08ca2c4e WatchSource:0}: Error finding container 614e54ef21056f0539ceff1274cde2abe4ab832020f2409785df4ebf08ca2c4e: Status 404 returned error can't find the container with id 614e54ef21056f0539ceff1274cde2abe4ab832020f2409785df4ebf08ca2c4e
Oct 30 21:00:28.724965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.724926    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd" event={"ID":"ffc30df6-d7dc-437e-9bba-a1053463abea","Type":"ContainerStarted","Data":"c70ba651faa79618f3c13ee2806027d494e99852f6b695d7f02c5adf401c9da4"}
Oct 30 21:00:28.725356 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.724989    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd" event={"ID":"ffc30df6-d7dc-437e-9bba-a1053463abea","Type":"ContainerStarted","Data":"614e54ef21056f0539ceff1274cde2abe4ab832020f2409785df4ebf08ca2c4e"}
Oct 30 21:00:28.742813 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.742766    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd" podStartSLOduration=1.742752903 podStartE2EDuration="1.742752903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:28.739561424 +0000 UTC m=+1395.231377970" watchObservedRunningTime="2026-10-30 21:00:28.742752903 +0000 UTC m=+1395.234569447"
Oct 30 21:00:28.895093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.895063    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"]
Oct 30 21:00:28.895680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.895660    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerName="cleanup"
Oct 30 21:00:28.895750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.895682    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7dff60-eef4-42ac-863d-7f8de225ead2" containerName="cleanup"
Oct 30 21:00:28.895750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.895722    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerName="cleanup"
Oct 30 21:00:28.895750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.895727    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae018d7-4f7d-4fb6-a441-f3f9785f2873" containerName="cleanup"
Oct 30 21:00:28.899422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.899405    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:28.909759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.908326    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"]
Oct 30 21:00:28.990344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.990314    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66509f8e-0592-44a8-b268-d33317745944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c31b208-fn2ct\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:28.990530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.990363    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66509f8e-0592-44a8-b268-d33317745944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c31b208-fn2ct\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:28.990530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:28.990386    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ffx\" (UniqueName: \"kubernetes.io/projected/66509f8e-0592-44a8-b268-d33317745944-kube-api-access-56ffx\") pod \"maas-api-revoke-sub-9c31b208-fn2ct\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:29.091930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.091892    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66509f8e-0592-44a8-b268-d33317745944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c31b208-fn2ct\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:29.092106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.091974    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66509f8e-0592-44a8-b268-d33317745944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c31b208-fn2ct\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:29.092106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.092008    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ffx\" (UniqueName: \"kubernetes.io/projected/66509f8e-0592-44a8-b268-d33317745944-kube-api-access-56ffx\") pod \"maas-api-revoke-sub-9c31b208-fn2ct\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:29.092626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.092602    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66509f8e-0592-44a8-b268-d33317745944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c31b208-fn2ct\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:29.094475 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.094443    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66509f8e-0592-44a8-b268-d33317745944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c31b208-fn2ct\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:29.101021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.100994    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ffx\" (UniqueName: \"kubernetes.io/projected/66509f8e-0592-44a8-b268-d33317745944-kube-api-access-56ffx\") pod \"maas-api-revoke-sub-9c31b208-fn2ct\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:29.226719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.226689    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:29.479633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.469898    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"]
Oct 30 21:00:29.527645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.527189    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"]
Oct 30 21:00:29.527645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.527291    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:29.534507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.534476    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"]
Oct 30 21:00:29.609989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.609953    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8871d183-kwhdr\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:29.610137 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.610091    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wscrn\" (UniqueName: \"kubernetes.io/projected/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-kube-api-access-wscrn\") pod \"maas-api-revoke-sub-8871d183-kwhdr\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:29.610368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.610337    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8871d183-kwhdr\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:29.716552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.716502    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wscrn\" (UniqueName: \"kubernetes.io/projected/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-kube-api-access-wscrn\") pod \"maas-api-revoke-sub-8871d183-kwhdr\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:29.716769 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.716748    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8871d183-kwhdr\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:29.716839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.716794    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8871d183-kwhdr\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:29.717952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.717922    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8871d183-kwhdr\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:29.725666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.725638    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8871d183-kwhdr\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:29.726272 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.726251    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wscrn\" (UniqueName: \"kubernetes.io/projected/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-kube-api-access-wscrn\") pod \"maas-api-revoke-sub-8871d183-kwhdr\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:29.736359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.736253    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct" event={"ID":"66509f8e-0592-44a8-b268-d33317745944","Type":"ContainerStarted","Data":"0a66de3949c18874ef417ef0366d7e69dfcde8aad2cd062968bc357d6f1c2984"}
Oct 30 21:00:29.738317 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.738286    4810 generic.go:356] "Generic (PLEG): container finished" podID="ffc30df6-d7dc-437e-9bba-a1053463abea" containerID="c70ba651faa79618f3c13ee2806027d494e99852f6b695d7f02c5adf401c9da4" exitCode=0
Oct 30 21:00:29.738429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.738360    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd" event={"ID":"ffc30df6-d7dc-437e-9bba-a1053463abea","Type":"ContainerDied","Data":"c70ba651faa79618f3c13ee2806027d494e99852f6b695d7f02c5adf401c9da4"}
Oct 30 21:00:29.855392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:29.855366    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:30.138551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:30.138527    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"]
Oct 30 21:00:30.140926 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:30.140885    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b54b68_d9a3_4fa4_ba54_ac6a47eb1d8b.slice/crio-8a18bcb5dbb43c7575dfb9242f1d9fece4fa9dd48f19a371a905a94c55fc4867 WatchSource:0}: Error finding container 8a18bcb5dbb43c7575dfb9242f1d9fece4fa9dd48f19a371a905a94c55fc4867: Status 404 returned error can't find the container with id 8a18bcb5dbb43c7575dfb9242f1d9fece4fa9dd48f19a371a905a94c55fc4867
Oct 30 21:00:30.749703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:30.749617    4810 generic.go:356] "Generic (PLEG): container finished" podID="66509f8e-0592-44a8-b268-d33317745944" containerID="c9af7784341c8292439029d7dd8dc4c001662d32be326d23b807a60da8d91d8d" exitCode=0
Oct 30 21:00:30.749703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:30.749658    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct" event={"ID":"66509f8e-0592-44a8-b268-d33317745944","Type":"ContainerDied","Data":"c9af7784341c8292439029d7dd8dc4c001662d32be326d23b807a60da8d91d8d"}
Oct 30 21:00:30.751500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:30.751469    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr" event={"ID":"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b","Type":"ContainerStarted","Data":"08e4fee36feaf8d9ea80db8ce22bc608a7e222077d5a208a0116a25a1255321d"}
Oct 30 21:00:30.751617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:30.751524    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr" event={"ID":"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b","Type":"ContainerStarted","Data":"8a18bcb5dbb43c7575dfb9242f1d9fece4fa9dd48f19a371a905a94c55fc4867"}
Oct 30 21:00:30.784658 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:30.784593    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr" podStartSLOduration=1.7845742900000001 podStartE2EDuration="1.78457429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:30.781132331 +0000 UTC m=+1397.272948874" watchObservedRunningTime="2026-10-30 21:00:30.78457429 +0000 UTC m=+1397.276390836"
Oct 30 21:00:31.292895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.276445    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:31.317332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.317303    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-openshift-ca\") pod \"ffc30df6-d7dc-437e-9bba-a1053463abea\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") "
Oct 30 21:00:31.318123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.318098    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc30df6-d7dc-437e-9bba-a1053463abea-kube-api-access-7gvb2\" (UniqueName: \"kubernetes.io/projected/ffc30df6-d7dc-437e-9bba-a1053463abea-kube-api-access-7gvb2\") pod \"ffc30df6-d7dc-437e-9bba-a1053463abea\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") "
Oct 30 21:00:31.318680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.318662    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-xks-ca\") pod \"ffc30df6-d7dc-437e-9bba-a1053463abea\" (UID: \"ffc30df6-d7dc-437e-9bba-a1053463abea\") "
Oct 30 21:00:31.319155 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.317912    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-openshift-ca" pod "ffc30df6-d7dc-437e-9bba-a1053463abea" (UID: "ffc30df6-d7dc-437e-9bba-a1053463abea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:31.327885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.319826    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.347074 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.346189    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc30df6-d7dc-437e-9bba-a1053463abea-kube-api-access-7gvb2" pod "ffc30df6-d7dc-437e-9bba-a1053463abea" (UID: "ffc30df6-d7dc-437e-9bba-a1053463abea"). InnerVolumeSpecName "kube-api-access-7gvb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:31.347074 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.346528    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-xks-ca" pod "ffc30df6-d7dc-437e-9bba-a1053463abea" (UID: "ffc30df6-d7dc-437e-9bba-a1053463abea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:31.459651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.435468    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc30df6-d7dc-437e-9bba-a1053463abea-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.459651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.435508    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gvb2\" (UniqueName: \"kubernetes.io/projected/ffc30df6-d7dc-437e-9bba-a1053463abea-kube-api-access-7gvb2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.876424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.876341    4810 generic.go:356] "Generic (PLEG): container finished" podID="13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b" containerID="08e4fee36feaf8d9ea80db8ce22bc608a7e222077d5a208a0116a25a1255321d" exitCode=0
Oct 30 21:00:31.884255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.882096    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr" event={"ID":"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b","Type":"ContainerDied","Data":"08e4fee36feaf8d9ea80db8ce22bc608a7e222077d5a208a0116a25a1255321d"}
Oct 30 21:00:31.930889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.919407    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"
Oct 30 21:00:31.930889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.919989    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd" event={"ID":"ffc30df6-d7dc-437e-9bba-a1053463abea","Type":"ContainerDied","Data":"614e54ef21056f0539ceff1274cde2abe4ab832020f2409785df4ebf08ca2c4e"}
Oct 30 21:00:31.930889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:31.920036    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="614e54ef21056f0539ceff1274cde2abe4ab832020f2409785df4ebf08ca2c4e"
Oct 30 21:00:32.444901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:32.444785    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:32.472019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:32.471971    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:32.805937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:32.804964    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:32.811337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:32.811284    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:33.167919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:33.157791    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:33.497285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:33.470075    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:33.497285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:33.470683    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 21:00:33.498496 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:33.497671    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:33.498496 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:33.498060    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 21:00:33.832882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:33.821925    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 21:00:33.832882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:33.821959    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:00:34.175883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.171250    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:34.215309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.215274    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 21:00:34.215806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.215788    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 21:00:34.216050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.216035    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 21:00:34.219884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.216799    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:34.444945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.443840    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66509f8e-0592-44a8-b268-d33317745944-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66509f8e-0592-44a8-b268-d33317745944-maas-api-xks-ca\") pod \"66509f8e-0592-44a8-b268-d33317745944\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") "
Oct 30 21:00:34.444945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.444046    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66509f8e-0592-44a8-b268-d33317745944-kube-api-access-56ffx\" (UniqueName: \"kubernetes.io/projected/66509f8e-0592-44a8-b268-d33317745944-kube-api-access-56ffx\") pod \"66509f8e-0592-44a8-b268-d33317745944\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") "
Oct 30 21:00:34.444945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.444109    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66509f8e-0592-44a8-b268-d33317745944-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66509f8e-0592-44a8-b268-d33317745944-maas-api-openshift-ca\") pod \"66509f8e-0592-44a8-b268-d33317745944\" (UID: \"66509f8e-0592-44a8-b268-d33317745944\") "
Oct 30 21:00:34.473316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.470946    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:34.473316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.472034    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66509f8e-0592-44a8-b268-d33317745944-maas-api-xks-ca" pod "66509f8e-0592-44a8-b268-d33317745944" (UID: "66509f8e-0592-44a8-b268-d33317745944"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:34.475320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.475272    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66509f8e-0592-44a8-b268-d33317745944-maas-api-openshift-ca" pod "66509f8e-0592-44a8-b268-d33317745944" (UID: "66509f8e-0592-44a8-b268-d33317745944"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:34.500414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.500370    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:34.615128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.608122    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66509f8e-0592-44a8-b268-d33317745944-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:34.615128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.608178    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66509f8e-0592-44a8-b268-d33317745944-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:34.852307 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.852217    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:34.854301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.854272    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:34.882668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.882567    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 21:00:34.915834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.915804    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"
Oct 30 21:00:34.918834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.918622    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct" event={"ID":"66509f8e-0592-44a8-b268-d33317745944","Type":"ContainerDied","Data":"0a66de3949c18874ef417ef0366d7e69dfcde8aad2cd062968bc357d6f1c2984"}
Oct 30 21:00:34.919048 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.919034    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a66de3949c18874ef417ef0366d7e69dfcde8aad2cd062968bc357d6f1c2984"
Oct 30 21:00:34.919173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:34.919163    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 21:00:35.344881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.270973    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:00:35.344881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.272694    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66509f8e-0592-44a8-b268-d33317745944-kube-api-access-56ffx" pod "66509f8e-0592-44a8-b268-d33317745944" (UID: "66509f8e-0592-44a8-b268-d33317745944"). InnerVolumeSpecName "kube-api-access-56ffx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:35.344881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.340354    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:00:35.344881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.340412    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:35.397663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.363802    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56ffx\" (UniqueName: \"kubernetes.io/projected/66509f8e-0592-44a8-b268-d33317745944-kube-api-access-56ffx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:35.601882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.530022    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:35.601882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.561371    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 21:00:35.601882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.574314    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 21:00:35.618990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.612561    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:35.646613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.646587    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 21:00:35.795243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.795159    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-kube-api-access-wscrn\" (UniqueName: \"kubernetes.io/projected/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-kube-api-access-wscrn\") pod \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") "
Oct 30 21:00:35.795447 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.795428    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-xks-ca\") pod \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") "
Oct 30 21:00:35.795519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.795482    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-openshift-ca\") pod \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\" (UID: \"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b\") "
Oct 30 21:00:35.796127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.796094    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-openshift-ca" pod "13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b" (UID: "13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:35.798103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.798076    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-kube-api-access-wscrn" pod "13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b" (UID: "13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b"). InnerVolumeSpecName "kube-api-access-wscrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:35.802638 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.802610    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-xks-ca" pod "13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b" (UID: "13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:35.898133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.898100    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:35.898133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.898136    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:35.898333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.898150    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wscrn\" (UniqueName: \"kubernetes.io/projected/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b-kube-api-access-wscrn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:35.923719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.923669    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"
Oct 30 21:00:35.923719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.923685    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr" event={"ID":"13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b","Type":"ContainerDied","Data":"8a18bcb5dbb43c7575dfb9242f1d9fece4fa9dd48f19a371a905a94c55fc4867"}
Oct 30 21:00:35.923934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:35.923754    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a18bcb5dbb43c7575dfb9242f1d9fece4fa9dd48f19a371a905a94c55fc4867"
Oct 30 21:00:39.832432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.832402    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"]
Oct 30 21:00:39.833051 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.833037    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ffc30df6-d7dc-437e-9bba-a1053463abea" containerName="revoke-keys"
Oct 30 21:00:39.833106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.833052    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc30df6-d7dc-437e-9bba-a1053463abea" containerName="revoke-keys"
Oct 30 21:00:39.833106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.833068    4810 cpu_manager.go:414] "containerMap: removing container" podUID="66509f8e-0592-44a8-b268-d33317745944" containerName="revoke-keys"
Oct 30 21:00:39.833106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.833073    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66509f8e-0592-44a8-b268-d33317745944" containerName="revoke-keys"
Oct 30 21:00:39.833106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.833086    4810 cpu_manager.go:414] "containerMap: removing container" podUID="13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b" containerName="revoke-keys"
Oct 30 21:00:39.833106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.833091    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b" containerName="revoke-keys"
Oct 30 21:00:39.833251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.833182    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc30df6-d7dc-437e-9bba-a1053463abea" containerName="revoke-keys"
Oct 30 21:00:39.833251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.833190    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="66509f8e-0592-44a8-b268-d33317745944" containerName="revoke-keys"
Oct 30 21:00:39.833251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.833198    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b" containerName="revoke-keys"
Oct 30 21:00:39.836768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.836752    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:39.841828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.840917    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:39.841828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.841305    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:00:39.844369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.844339    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"]
Oct 30 21:00:39.850144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.850115    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6bfc82-7wmpw\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:39.850386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.850368    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6bfc82-7wmpw\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:39.850496 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.850477    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbvz\" (UniqueName: \"kubernetes.io/projected/250cf721-68c9-42de-b624-7d1a930b4867-kube-api-access-zqbvz\") pod \"maas-api-revoke-sub-3d6bfc82-7wmpw\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:39.951275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.951247    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6bfc82-7wmpw\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:39.951455 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.951302    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbvz\" (UniqueName: \"kubernetes.io/projected/250cf721-68c9-42de-b624-7d1a930b4867-kube-api-access-zqbvz\") pod \"maas-api-revoke-sub-3d6bfc82-7wmpw\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:39.951455 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.951356    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6bfc82-7wmpw\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:39.952139 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.952118    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6bfc82-7wmpw\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:39.953765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.953737    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6bfc82-7wmpw\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:39.960636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:39.960616    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbvz\" (UniqueName: \"kubernetes.io/projected/250cf721-68c9-42de-b624-7d1a930b4867-kube-api-access-zqbvz\") pod \"maas-api-revoke-sub-3d6bfc82-7wmpw\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:40.153703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:40.153665    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:40.454400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:40.454365    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"]
Oct 30 21:00:40.455261 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:40.455238    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250cf721_68c9_42de_b624_7d1a930b4867.slice/crio-1f6ddc9f94d22f1ff8b3fb691de7f8047e2608e96566a835bda5dbed2f7dac6e WatchSource:0}: Error finding container 1f6ddc9f94d22f1ff8b3fb691de7f8047e2608e96566a835bda5dbed2f7dac6e: Status 404 returned error can't find the container with id 1f6ddc9f94d22f1ff8b3fb691de7f8047e2608e96566a835bda5dbed2f7dac6e
Oct 30 21:00:40.961808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:40.961774    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw" event={"ID":"250cf721-68c9-42de-b624-7d1a930b4867","Type":"ContainerStarted","Data":"057b7761e48df49e1c67014a842db7cfba39f760ebb5e9f57b197f4813866f5f"}
Oct 30 21:00:40.962261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:40.961824    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw" event={"ID":"250cf721-68c9-42de-b624-7d1a930b4867","Type":"ContainerStarted","Data":"1f6ddc9f94d22f1ff8b3fb691de7f8047e2608e96566a835bda5dbed2f7dac6e"}
Oct 30 21:00:40.988655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:40.988609    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw" podStartSLOduration=1.9885930840000001 podStartE2EDuration="1.988593084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:40.985749988 +0000 UTC m=+1407.477566532" watchObservedRunningTime="2026-10-30 21:00:40.988593084 +0000 UTC m=+1407.480409633"
Oct 30 21:00:41.287555 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.287520    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"]
Oct 30 21:00:41.293514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.293488    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.329233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.328656    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"]
Oct 30 21:00:41.373138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.370198    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d1af73b-2jdb2\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.373138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.370249    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d1af73b-2jdb2\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.373138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.370398    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79962\" (UniqueName: \"kubernetes.io/projected/7d225e5d-b49e-4b42-b81d-e1bf519349af-kube-api-access-79962\") pod \"maas-api-revoke-sub-4d1af73b-2jdb2\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.472558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.472518    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d1af73b-2jdb2\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.472558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.472560    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d1af73b-2jdb2\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.472810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.472613    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79962\" (UniqueName: \"kubernetes.io/projected/7d225e5d-b49e-4b42-b81d-e1bf519349af-kube-api-access-79962\") pod \"maas-api-revoke-sub-4d1af73b-2jdb2\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.473539 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.473517    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d1af73b-2jdb2\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.476014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.475990    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d1af73b-2jdb2\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.482256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.482129    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79962\" (UniqueName: \"kubernetes.io/projected/7d225e5d-b49e-4b42-b81d-e1bf519349af-kube-api-access-79962\") pod \"maas-api-revoke-sub-4d1af73b-2jdb2\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.634887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.634116    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:41.974206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.974132    4810 generic.go:356] "Generic (PLEG): container finished" podID="250cf721-68c9-42de-b624-7d1a930b4867" containerID="057b7761e48df49e1c67014a842db7cfba39f760ebb5e9f57b197f4813866f5f" exitCode=0
Oct 30 21:00:41.974632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.974209    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw" event={"ID":"250cf721-68c9-42de-b624-7d1a930b4867","Type":"ContainerDied","Data":"057b7761e48df49e1c67014a842db7cfba39f760ebb5e9f57b197f4813866f5f"}
Oct 30 21:00:41.987036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:41.987011    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"]
Oct 30 21:00:41.987619 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:41.987594    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d225e5d_b49e_4b42_b81d_e1bf519349af.slice/crio-a3ac798a81d25ca5dd5e411d89fc3ac3e65f81daad5cee5ac4713d832b6d1dc6 WatchSource:0}: Error finding container a3ac798a81d25ca5dd5e411d89fc3ac3e65f81daad5cee5ac4713d832b6d1dc6: Status 404 returned error can't find the container with id a3ac798a81d25ca5dd5e411d89fc3ac3e65f81daad5cee5ac4713d832b6d1dc6
Oct 30 21:00:42.060351 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.060323    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"]
Oct 30 21:00:42.065810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.065791    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.078900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.076978    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"]
Oct 30 21:00:42.187884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.187823    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a3ee10-kbkj7\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.188100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.188074    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8cb\" (UniqueName: \"kubernetes.io/projected/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-kube-api-access-9z8cb\") pod \"maas-api-revoke-sub-91a3ee10-kbkj7\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.188226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.188207    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a3ee10-kbkj7\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.290395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.290366    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8cb\" (UniqueName: \"kubernetes.io/projected/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-kube-api-access-9z8cb\") pod \"maas-api-revoke-sub-91a3ee10-kbkj7\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.290551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.290470    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a3ee10-kbkj7\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.290613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.290568    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a3ee10-kbkj7\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.291327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.291304    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91a3ee10-kbkj7\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.293178 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.293152    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91a3ee10-kbkj7\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.298955 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.298933    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8cb\" (UniqueName: \"kubernetes.io/projected/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-kube-api-access-9z8cb\") pod \"maas-api-revoke-sub-91a3ee10-kbkj7\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.394307 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.394273    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:42.446378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.445428    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"]
Oct 30 21:00:42.446378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.445643    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g" podUID="3a820c1c-f9f6-4c54-a8eb-423f012d340e" containerName="payload-processing" containerID="cri-o://e788547cac81a64dece86ea2f9a33fe0e8df1594d0dd3b6fd3f039a27274f77a" gracePeriod=30
Oct 30 21:00:42.511087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.510273    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"]
Oct 30 21:00:42.511087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.510507    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz" podUID="46a7a59d-61be-4175-ad13-d5a5d33fe297" containerName="payload-pre-processing" containerID="cri-o://e7b63ff260f245fd70fdfa24710c702b9554e712feab5c384c907d5712b1c9c6" gracePeriod=30
Oct 30 21:00:42.558874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.553755    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:00:42.959843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.959793    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"]
Oct 30 21:00:42.998355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.998321    4810 generic.go:356] "Generic (PLEG): container finished" podID="46a7a59d-61be-4175-ad13-d5a5d33fe297" containerID="e7b63ff260f245fd70fdfa24710c702b9554e712feab5c384c907d5712b1c9c6" exitCode=0
Oct 30 21:00:42.999006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:42.998977    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz" event={"ID":"46a7a59d-61be-4175-ad13-d5a5d33fe297","Type":"ContainerDied","Data":"e7b63ff260f245fd70fdfa24710c702b9554e712feab5c384c907d5712b1c9c6"}
Oct 30 21:00:43.001547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.001521    4810 generic.go:356] "Generic (PLEG): container finished" podID="3a820c1c-f9f6-4c54-a8eb-423f012d340e" containerID="e788547cac81a64dece86ea2f9a33fe0e8df1594d0dd3b6fd3f039a27274f77a" exitCode=0
Oct 30 21:00:43.001874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.001836    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g" event={"ID":"3a820c1c-f9f6-4c54-a8eb-423f012d340e","Type":"ContainerDied","Data":"e788547cac81a64dece86ea2f9a33fe0e8df1594d0dd3b6fd3f039a27274f77a"}
Oct 30 21:00:43.004495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.004330    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2" event={"ID":"7d225e5d-b49e-4b42-b81d-e1bf519349af","Type":"ContainerStarted","Data":"ff792dff393d7f62760b7dfec6f44767483ea737b3206082759bdae103f8b5b2"}
Oct 30 21:00:43.004698 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.004677    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2" event={"ID":"7d225e5d-b49e-4b42-b81d-e1bf519349af","Type":"ContainerStarted","Data":"a3ac798a81d25ca5dd5e411d89fc3ac3e65f81daad5cee5ac4713d832b6d1dc6"}
Oct 30 21:00:43.044119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.044071    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2" podStartSLOduration=2.044056881 podStartE2EDuration="2.044056881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:43.038334801 +0000 UTC m=+1409.530151344" watchObservedRunningTime="2026-10-30 21:00:43.044056881 +0000 UTC m=+1409.535873426"
Oct 30 21:00:43.073354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.073326    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"]
Oct 30 21:00:43.085495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.085469    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-955eb7-mxf7p"]
Oct 30 21:00:43.109050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.108009    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 21:00:43.183344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.182380    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a820c1c-f9f6-4c54-a8eb-423f012d340e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a820c1c-f9f6-4c54-a8eb-423f012d340e-plugins-config-volume\") pod \"3a820c1c-f9f6-4c54-a8eb-423f012d340e\" (UID: \"3a820c1c-f9f6-4c54-a8eb-423f012d340e\") "
Oct 30 21:00:43.183344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.182427    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a820c1c-f9f6-4c54-a8eb-423f012d340e-kube-api-access-lrqxd\" (UniqueName: \"kubernetes.io/projected/3a820c1c-f9f6-4c54-a8eb-423f012d340e-kube-api-access-lrqxd\") pod \"3a820c1c-f9f6-4c54-a8eb-423f012d340e\" (UID: \"3a820c1c-f9f6-4c54-a8eb-423f012d340e\") "
Oct 30 21:00:43.184232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.184207    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a820c1c-f9f6-4c54-a8eb-423f012d340e-plugins-config-volume" pod "3a820c1c-f9f6-4c54-a8eb-423f012d340e" (UID: "3a820c1c-f9f6-4c54-a8eb-423f012d340e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:43.196919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.196880    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 21:00:43.281627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.273049    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a820c1c-f9f6-4c54-a8eb-423f012d340e-kube-api-access-lrqxd" pod "3a820c1c-f9f6-4c54-a8eb-423f012d340e" (UID: "3a820c1c-f9f6-4c54-a8eb-423f012d340e"). InnerVolumeSpecName "kube-api-access-lrqxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:43.283456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.283431    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a7a59d-61be-4175-ad13-d5a5d33fe297-kube-api-access-lgh96\" (UniqueName: \"kubernetes.io/projected/46a7a59d-61be-4175-ad13-d5a5d33fe297-kube-api-access-lgh96\") pod \"46a7a59d-61be-4175-ad13-d5a5d33fe297\" (UID: \"46a7a59d-61be-4175-ad13-d5a5d33fe297\") "
Oct 30 21:00:43.283643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.283624    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a7a59d-61be-4175-ad13-d5a5d33fe297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a7a59d-61be-4175-ad13-d5a5d33fe297-plugins-config-volume\") pod \"46a7a59d-61be-4175-ad13-d5a5d33fe297\" (UID: \"46a7a59d-61be-4175-ad13-d5a5d33fe297\") "
Oct 30 21:00:43.284161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.284140    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a820c1c-f9f6-4c54-a8eb-423f012d340e-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:43.284161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.284149    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a7a59d-61be-4175-ad13-d5a5d33fe297-plugins-config-volume" pod "46a7a59d-61be-4175-ad13-d5a5d33fe297" (UID: "46a7a59d-61be-4175-ad13-d5a5d33fe297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:43.284316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.284163    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqxd\" (UniqueName: \"kubernetes.io/projected/3a820c1c-f9f6-4c54-a8eb-423f012d340e-kube-api-access-lrqxd\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:43.286176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.286146    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a7a59d-61be-4175-ad13-d5a5d33fe297-kube-api-access-lgh96" pod "46a7a59d-61be-4175-ad13-d5a5d33fe297" (UID: "46a7a59d-61be-4175-ad13-d5a5d33fe297"). InnerVolumeSpecName "kube-api-access-lgh96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:43.337060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.337023    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:43.385817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.385787    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a7a59d-61be-4175-ad13-d5a5d33fe297-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:43.385817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.385813    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgh96\" (UniqueName: \"kubernetes.io/projected/46a7a59d-61be-4175-ad13-d5a5d33fe297-kube-api-access-lgh96\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:43.490395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.490310    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-openshift-ca\") pod \"250cf721-68c9-42de-b624-7d1a930b4867\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") "
Oct 30 21:00:43.490768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.490611    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-xks-ca\") pod \"250cf721-68c9-42de-b624-7d1a930b4867\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") "
Oct 30 21:00:43.490768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.490677    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250cf721-68c9-42de-b624-7d1a930b4867-kube-api-access-zqbvz\" (UniqueName: \"kubernetes.io/projected/250cf721-68c9-42de-b624-7d1a930b4867-kube-api-access-zqbvz\") pod \"250cf721-68c9-42de-b624-7d1a930b4867\" (UID: \"250cf721-68c9-42de-b624-7d1a930b4867\") "
Oct 30 21:00:43.491112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.491077    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-openshift-ca" pod "250cf721-68c9-42de-b624-7d1a930b4867" (UID: "250cf721-68c9-42de-b624-7d1a930b4867"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:43.494162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.494137    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-xks-ca" pod "250cf721-68c9-42de-b624-7d1a930b4867" (UID: "250cf721-68c9-42de-b624-7d1a930b4867"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:43.494378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.494350    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250cf721-68c9-42de-b624-7d1a930b4867-kube-api-access-zqbvz" pod "250cf721-68c9-42de-b624-7d1a930b4867" (UID: "250cf721-68c9-42de-b624-7d1a930b4867"). InnerVolumeSpecName "kube-api-access-zqbvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:43.592489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.592458    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:43.592630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.592492    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqbvz\" (UniqueName: \"kubernetes.io/projected/250cf721-68c9-42de-b624-7d1a930b4867-kube-api-access-zqbvz\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:43.592630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.592509    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250cf721-68c9-42de-b624-7d1a930b4867-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:43.759931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:43.759828    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eac8948-5a0d-46e0-ab36-38ae2cff51bf" path="/var/lib/kubelet/pods/5eac8948-5a0d-46e0-ab36-38ae2cff51bf/volumes"
Oct 30 21:00:44.155937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.145358    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz" event={"ID":"46a7a59d-61be-4175-ad13-d5a5d33fe297","Type":"ContainerDied","Data":"9904e53788204093e72f22718e3ff457509f39696c8dda16fcabc3eb6ccecb34"}
Oct 30 21:00:44.155937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.145486    4810 scope.go:122] "RemoveContainer" containerID="e7b63ff260f245fd70fdfa24710c702b9554e712feab5c384c907d5712b1c9c6"
Oct 30 21:00:44.155937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.145695    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"
Oct 30 21:00:44.242829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.233647    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g" event={"ID":"3a820c1c-f9f6-4c54-a8eb-423f012d340e","Type":"ContainerDied","Data":"727e2d992b2a267a26cce42db82d89b6b4ad86207cc82ee25283974ec573d5a0"}
Oct 30 21:00:44.242829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.234454    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"
Oct 30 21:00:44.265649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.265598    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7" event={"ID":"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6","Type":"ContainerStarted","Data":"4b97f058926bb4c8493c3a1ecbf220c26927bc18bdbfc4d380febe3543b4af2f"}
Oct 30 21:00:44.270005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.269973    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7" event={"ID":"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6","Type":"ContainerStarted","Data":"42afecd85818d5927c083a9c1d98eef16d42a029128b22f90a25553abc25c1dc"}
Oct 30 21:00:44.429930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.406577    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"
Oct 30 21:00:44.429930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.419607    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"]
Oct 30 21:00:44.429930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.419653    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw" event={"ID":"250cf721-68c9-42de-b624-7d1a930b4867","Type":"ContainerDied","Data":"1f6ddc9f94d22f1ff8b3fb691de7f8047e2608e96566a835bda5dbed2f7dac6e"}
Oct 30 21:00:44.429930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.422965    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f6ddc9f94d22f1ff8b3fb691de7f8047e2608e96566a835bda5dbed2f7dac6e"
Oct 30 21:00:44.515959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.472083    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-656c7cf559-vcxmz"]
Oct 30 21:00:44.554150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.554123    4810 scope.go:122] "RemoveContainer" containerID="e788547cac81a64dece86ea2f9a33fe0e8df1594d0dd3b6fd3f039a27274f77a"
Oct 30 21:00:44.557963 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.557930    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2" event={"ID":"7d225e5d-b49e-4b42-b81d-e1bf519349af","Type":"ContainerDied","Data":"ff792dff393d7f62760b7dfec6f44767483ea737b3206082759bdae103f8b5b2"}
Oct 30 21:00:44.558474 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.557134    4810 generic.go:356] "Generic (PLEG): container finished" podID="7d225e5d-b49e-4b42-b81d-e1bf519349af" containerID="ff792dff393d7f62760b7dfec6f44767483ea737b3206082759bdae103f8b5b2" exitCode=0
Oct 30 21:00:44.563066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.563047    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"]
Oct 30 21:00:44.633686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.633623    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7" podStartSLOduration=2.633602534 podStartE2EDuration="2.633602534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:44.487251246 +0000 UTC m=+1410.979067813" watchObservedRunningTime="2026-10-30 21:00:44.633602534 +0000 UTC m=+1411.125419080"
Oct 30 21:00:44.643550 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:44.643514    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-78b55758fb-cnj4g"]
Oct 30 21:00:45.658380 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:45.658344    4810 generic.go:356] "Generic (PLEG): container finished" podID="643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6" containerID="4b97f058926bb4c8493c3a1ecbf220c26927bc18bdbfc4d380febe3543b4af2f" exitCode=0
Oct 30 21:00:45.659562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:45.659533    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7" event={"ID":"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6","Type":"ContainerDied","Data":"4b97f058926bb4c8493c3a1ecbf220c26927bc18bdbfc4d380febe3543b4af2f"}
Oct 30 21:00:45.753948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:45.753913    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a820c1c-f9f6-4c54-a8eb-423f012d340e" path="/var/lib/kubelet/pods/3a820c1c-f9f6-4c54-a8eb-423f012d340e/volumes"
Oct 30 21:00:45.787319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:45.787283    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a7a59d-61be-4175-ad13-d5a5d33fe297" path="/var/lib/kubelet/pods/46a7a59d-61be-4175-ad13-d5a5d33fe297/volumes"
Oct 30 21:00:45.894027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:45.893982    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:46.083940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:46.082981    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:46.491884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:46.478493    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:00:46.535061 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:46.535004    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:46.594041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:46.593998    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:46.924142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:46.919947    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:00:46.924142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:46.920431    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 21:00:47.500884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:47.481317    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:00:47.500884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:47.482888    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 21:00:47.551995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:47.551951    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:47.578930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:47.553817    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 21:00:47.630926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:47.615275    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:47.630926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:47.621020    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 21:00:47.817881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:47.804124    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:47.851932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:47.851364    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:48.008577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.008528    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:48.077166 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.064680    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2" event={"ID":"7d225e5d-b49e-4b42-b81d-e1bf519349af","Type":"ContainerDied","Data":"a3ac798a81d25ca5dd5e411d89fc3ac3e65f81daad5cee5ac4713d832b6d1dc6"}
Oct 30 21:00:48.077166 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.064766    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ac798a81d25ca5dd5e411d89fc3ac3e65f81daad5cee5ac4713d832b6d1dc6"
Oct 30 21:00:48.077166 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.064903    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"
Oct 30 21:00:48.226115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.169161    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-openshift-ca\") pod \"7d225e5d-b49e-4b42-b81d-e1bf519349af\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") "
Oct 30 21:00:48.226115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.169364    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-xks-ca\") pod \"7d225e5d-b49e-4b42-b81d-e1bf519349af\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") "
Oct 30 21:00:48.226115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.169493    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d225e5d-b49e-4b42-b81d-e1bf519349af-kube-api-access-79962\" (UniqueName: \"kubernetes.io/projected/7d225e5d-b49e-4b42-b81d-e1bf519349af-kube-api-access-79962\") pod \"7d225e5d-b49e-4b42-b81d-e1bf519349af\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") "
Oct 30 21:00:48.239908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.239847    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-xks-ca" pod "7d225e5d-b49e-4b42-b81d-e1bf519349af" (UID: "7d225e5d-b49e-4b42-b81d-e1bf519349af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:48.297894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.282954    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:00:48.326881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.322509    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-openshift-ca" pod "7d225e5d-b49e-4b42-b81d-e1bf519349af" (UID: "7d225e5d-b49e-4b42-b81d-e1bf519349af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:48.331885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.327277    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-openshift-ca\") pod \"7d225e5d-b49e-4b42-b81d-e1bf519349af\" (UID: \"7d225e5d-b49e-4b42-b81d-e1bf519349af\") "
Oct 30 21:00:48.413435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.413398    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:48.413659 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:48.335141    4810 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/7d225e5d-b49e-4b42-b81d-e1bf519349af/volumes/kubernetes.io~configmap/maas-api-openshift-ca
Oct 30 21:00:48.413783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.413758    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-openshift-ca" pod "7d225e5d-b49e-4b42-b81d-e1bf519349af" (UID: "7d225e5d-b49e-4b42-b81d-e1bf519349af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:48.470722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.470647    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:00:48.471049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.471002    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:48.492521 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.492413    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:48.558798 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.558747    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:48.562015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.561990    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d225e5d-b49e-4b42-b81d-e1bf519349af-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:48.593893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.589505    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d225e5d-b49e-4b42-b81d-e1bf519349af-kube-api-access-79962" pod "7d225e5d-b49e-4b42-b81d-e1bf519349af" (UID: "7d225e5d-b49e-4b42-b81d-e1bf519349af"). InnerVolumeSpecName "kube-api-access-79962". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:48.624354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.624312    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:48.642022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.628614    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 21:00:48.693649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.693621    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79962\" (UniqueName: \"kubernetes.io/projected/7d225e5d-b49e-4b42-b81d-e1bf519349af-kube-api-access-79962\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:48.745336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.744708    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 21:00:48.761656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.760156    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 21:00:48.762105 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.762070    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 21:00:48.782403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.782373    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:48.799906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.799542    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-kube-api-access-9z8cb\" (UniqueName: \"kubernetes.io/projected/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-kube-api-access-9z8cb\") pod \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") "
Oct 30 21:00:48.799906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.799624    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-openshift-ca\") pod \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") "
Oct 30 21:00:48.799906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.799730    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-xks-ca\") pod \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\" (UID: \"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6\") "
Oct 30 21:00:48.803202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.803175    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-openshift-ca" pod "643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6" (UID: "643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:48.806977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.806929    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-xks-ca" pod "643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6" (UID: "643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:48.813401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.809658    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-kube-api-access-9z8cb" pod "643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6" (UID: "643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6"). InnerVolumeSpecName "kube-api-access-9z8cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:48.901592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.901560    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z8cb\" (UniqueName: \"kubernetes.io/projected/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-kube-api-access-9z8cb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:48.901592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.901594    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:48.901812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:48.901615    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:49.076344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:49.076268    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"
Oct 30 21:00:49.076739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:49.076267    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7" event={"ID":"643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6","Type":"ContainerDied","Data":"42afecd85818d5927c083a9c1d98eef16d42a029128b22f90a25553abc25c1dc"}
Oct 30 21:00:49.076739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:49.076416    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42afecd85818d5927c083a9c1d98eef16d42a029128b22f90a25553abc25c1dc"
Oct 30 21:00:51.512489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.512459    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"]
Oct 30 21:00:51.513077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513063    4810 cpu_manager.go:414] "containerMap: removing container" podUID="250cf721-68c9-42de-b624-7d1a930b4867" containerName="revoke-keys"
Oct 30 21:00:51.513125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513078    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250cf721-68c9-42de-b624-7d1a930b4867" containerName="revoke-keys"
Oct 30 21:00:51.513125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513101    4810 cpu_manager.go:414] "containerMap: removing container" podUID="46a7a59d-61be-4175-ad13-d5a5d33fe297" containerName="payload-pre-processing"
Oct 30 21:00:51.513125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513106    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a7a59d-61be-4175-ad13-d5a5d33fe297" containerName="payload-pre-processing"
Oct 30 21:00:51.513212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513126    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7d225e5d-b49e-4b42-b81d-e1bf519349af" containerName="revoke-keys"
Oct 30 21:00:51.513212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513132    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d225e5d-b49e-4b42-b81d-e1bf519349af" containerName="revoke-keys"
Oct 30 21:00:51.513212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513145    4810 cpu_manager.go:414] "containerMap: removing container" podUID="643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6" containerName="revoke-keys"
Oct 30 21:00:51.513212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513150    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6" containerName="revoke-keys"
Oct 30 21:00:51.513212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513163    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3a820c1c-f9f6-4c54-a8eb-423f012d340e" containerName="payload-processing"
Oct 30 21:00:51.513212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513168    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a820c1c-f9f6-4c54-a8eb-423f012d340e" containerName="payload-processing"
Oct 30 21:00:51.513389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513246    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a7a59d-61be-4175-ad13-d5a5d33fe297" containerName="payload-pre-processing"
Oct 30 21:00:51.513389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513258    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d225e5d-b49e-4b42-b81d-e1bf519349af" containerName="revoke-keys"
Oct 30 21:00:51.513389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513266    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6" containerName="revoke-keys"
Oct 30 21:00:51.513389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513272    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a820c1c-f9f6-4c54-a8eb-423f012d340e" containerName="payload-processing"
Oct 30 21:00:51.513389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.513279    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="250cf721-68c9-42de-b624-7d1a930b4867" containerName="revoke-keys"
Oct 30 21:00:51.517335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.517317    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:51.521791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.521765    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:00:51.521937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.521851    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:51.525802 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.525779    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"]
Oct 30 21:00:51.540155 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.540125    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:51.540324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.540249    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:51.540395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.540323    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szcq4\" (UniqueName: \"kubernetes.io/projected/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-kube-api-access-szcq4\") pod \"maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:51.642027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.641993    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:51.642205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.642104    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:51.642205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.642175    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szcq4\" (UniqueName: \"kubernetes.io/projected/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-kube-api-access-szcq4\") pod \"maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:51.642720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.642696    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:51.644378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.644354    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:51.656138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.656116    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szcq4\" (UniqueName: \"kubernetes.io/projected/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-kube-api-access-szcq4\") pod \"maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:51.835245 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:51.835167    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:52.133584 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:52.133538    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8a98c9_ab84_4b93_b7f8_8572784ed69f.slice/crio-cd39f6a0bb720640838341a7d554e374d84203ae843f7deea8361bfbb990ff28 WatchSource:0}: Error finding container cd39f6a0bb720640838341a7d554e374d84203ae843f7deea8361bfbb990ff28: Status 404 returned error can't find the container with id cd39f6a0bb720640838341a7d554e374d84203ae843f7deea8361bfbb990ff28
Oct 30 21:00:52.133891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:52.133833    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"]
Oct 30 21:00:53.119670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:53.119625    4810 generic.go:356] "Generic (PLEG): container finished" podID="6f8a98c9-ab84-4b93-b7f8-8572784ed69f" containerID="490d4a68cdaae90487f2116976a7a071c88e527f56ef0e5f60ec44d16b859f56" exitCode=0
Oct 30 21:00:53.120189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:53.119718    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh" event={"ID":"6f8a98c9-ab84-4b93-b7f8-8572784ed69f","Type":"ContainerDied","Data":"490d4a68cdaae90487f2116976a7a071c88e527f56ef0e5f60ec44d16b859f56"}
Oct 30 21:00:53.120189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:53.119765    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh" event={"ID":"6f8a98c9-ab84-4b93-b7f8-8572784ed69f","Type":"ContainerStarted","Data":"cd39f6a0bb720640838341a7d554e374d84203ae843f7deea8361bfbb990ff28"}
Oct 30 21:00:54.314923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:54.314852    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:54.379544 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:54.379447    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-xks-ca\") pod \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") "
Oct 30 21:00:54.379749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:54.379630    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-kube-api-access-szcq4\" (UniqueName: \"kubernetes.io/projected/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-kube-api-access-szcq4\") pod \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") "
Oct 30 21:00:54.379749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:54.379673    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-openshift-ca\") pod \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\" (UID: \"6f8a98c9-ab84-4b93-b7f8-8572784ed69f\") "
Oct 30 21:00:54.380665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:54.380636    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-openshift-ca" pod "6f8a98c9-ab84-4b93-b7f8-8572784ed69f" (UID: "6f8a98c9-ab84-4b93-b7f8-8572784ed69f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:54.388893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:54.386422    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-kube-api-access-szcq4" pod "6f8a98c9-ab84-4b93-b7f8-8572784ed69f" (UID: "6f8a98c9-ab84-4b93-b7f8-8572784ed69f"). InnerVolumeSpecName "kube-api-access-szcq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:54.394211 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:54.394169    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-xks-ca" pod "6f8a98c9-ab84-4b93-b7f8-8572784ed69f" (UID: "6f8a98c9-ab84-4b93-b7f8-8572784ed69f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:54.481799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:54.481759    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:54.481799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:54.481790    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szcq4\" (UniqueName: \"kubernetes.io/projected/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-kube-api-access-szcq4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:54.481799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:54.481805    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f8a98c9-ab84-4b93-b7f8-8572784ed69f-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.138931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:55.138892    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh" event={"ID":"6f8a98c9-ab84-4b93-b7f8-8572784ed69f","Type":"ContainerDied","Data":"cd39f6a0bb720640838341a7d554e374d84203ae843f7deea8361bfbb990ff28"}
Oct 30 21:00:55.139092 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:55.138954    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd39f6a0bb720640838341a7d554e374d84203ae843f7deea8361bfbb990ff28"
Oct 30 21:00:55.139092 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:55.138907    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"
Oct 30 21:00:55.158686 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:00:55.158655    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:00:56.446950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.446915    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"]
Oct 30 21:00:56.447518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.447505    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6f8a98c9-ab84-4b93-b7f8-8572784ed69f" containerName="revoke-keys"
Oct 30 21:00:56.447562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.447519    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8a98c9-ab84-4b93-b7f8-8572784ed69f" containerName="revoke-keys"
Oct 30 21:00:56.447638 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.447629    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8a98c9-ab84-4b93-b7f8-8572784ed69f" containerName="revoke-keys"
Oct 30 21:00:56.451392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.451367    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:56.463760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.463733    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:56.464261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.464243    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:00:56.464767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.464746    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"]
Oct 30 21:00:56.517953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.516310    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cca24a1-ctsc2\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:56.517953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.516399    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cca24a1-ctsc2\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:56.517953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.516800    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46c8m\" (UniqueName: \"kubernetes.io/projected/5ce3e833-d075-43d6-b477-774d5f680a1f-kube-api-access-46c8m\") pod \"maas-api-revoke-sub-1cca24a1-ctsc2\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:56.617779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.617742    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46c8m\" (UniqueName: \"kubernetes.io/projected/5ce3e833-d075-43d6-b477-774d5f680a1f-kube-api-access-46c8m\") pod \"maas-api-revoke-sub-1cca24a1-ctsc2\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:56.617995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.617826    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cca24a1-ctsc2\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:56.617995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.617875    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cca24a1-ctsc2\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:56.619059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.618664    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cca24a1-ctsc2\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:56.621280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.621239    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cca24a1-ctsc2\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:56.627158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.627132    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46c8m\" (UniqueName: \"kubernetes.io/projected/5ce3e833-d075-43d6-b477-774d5f680a1f-kube-api-access-46c8m\") pod \"maas-api-revoke-sub-1cca24a1-ctsc2\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:56.765954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:56.765844    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:57.051541 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:57.051505    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce3e833_d075_43d6_b477_774d5f680a1f.slice/crio-ff824ca12957e0958d799b3012bc0ecfa5d760d6720c7226b38cc822687ff958 WatchSource:0}: Error finding container ff824ca12957e0958d799b3012bc0ecfa5d760d6720c7226b38cc822687ff958: Status 404 returned error can't find the container with id ff824ca12957e0958d799b3012bc0ecfa5d760d6720c7226b38cc822687ff958
Oct 30 21:00:57.053173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:57.053147    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"]
Oct 30 21:00:57.155408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:57.155379    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2" event={"ID":"5ce3e833-d075-43d6-b477-774d5f680a1f","Type":"ContainerStarted","Data":"ff824ca12957e0958d799b3012bc0ecfa5d760d6720c7226b38cc822687ff958"}
Oct 30 21:00:58.164941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:58.164893    4810 generic.go:356] "Generic (PLEG): container finished" podID="5ce3e833-d075-43d6-b477-774d5f680a1f" containerID="b6150e1600eaa7b749cb64dd4fce2e0376b736ffbe342cbd648b993550739433" exitCode=0
Oct 30 21:00:58.165309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:58.164983    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2" event={"ID":"5ce3e833-d075-43d6-b477-774d5f680a1f","Type":"ContainerDied","Data":"b6150e1600eaa7b749cb64dd4fce2e0376b736ffbe342cbd648b993550739433"}
Oct 30 21:00:59.127504 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.127468    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"]
Oct 30 21:00:59.131817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.131790    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.210886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.210820    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"]
Oct 30 21:00:59.256935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.256720    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df431714-sl8kg\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.256935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.256766    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwjgg\" (UniqueName: \"kubernetes.io/projected/5871f831-513b-4c57-9be3-2b5b296dbe00-kube-api-access-wwjgg\") pod \"maas-api-revoke-sub-df431714-sl8kg\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.256935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.256814    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df431714-sl8kg\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.357679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.357652    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df431714-sl8kg\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.357836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.357703    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwjgg\" (UniqueName: \"kubernetes.io/projected/5871f831-513b-4c57-9be3-2b5b296dbe00-kube-api-access-wwjgg\") pod \"maas-api-revoke-sub-df431714-sl8kg\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.357960 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.357891    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df431714-sl8kg\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.358355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.358332    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df431714-sl8kg\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.360310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.360283    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df431714-sl8kg\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.361119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.361104    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:00:59.372234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.372215    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwjgg\" (UniqueName: \"kubernetes.io/projected/5871f831-513b-4c57-9be3-2b5b296dbe00-kube-api-access-wwjgg\") pod \"maas-api-revoke-sub-df431714-sl8kg\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.445657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.445620    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:00:59.459307 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.459276    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-xks-ca\") pod \"5ce3e833-d075-43d6-b477-774d5f680a1f\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") "
Oct 30 21:00:59.459442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.459340    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce3e833-d075-43d6-b477-774d5f680a1f-kube-api-access-46c8m\" (UniqueName: \"kubernetes.io/projected/5ce3e833-d075-43d6-b477-774d5f680a1f-kube-api-access-46c8m\") pod \"5ce3e833-d075-43d6-b477-774d5f680a1f\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") "
Oct 30 21:00:59.459512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.459442    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-openshift-ca\") pod \"5ce3e833-d075-43d6-b477-774d5f680a1f\" (UID: \"5ce3e833-d075-43d6-b477-774d5f680a1f\") "
Oct 30 21:00:59.459915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.459837    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-openshift-ca" pod "5ce3e833-d075-43d6-b477-774d5f680a1f" (UID: "5ce3e833-d075-43d6-b477-774d5f680a1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:59.460119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.460095    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:59.461658 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.461625    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce3e833-d075-43d6-b477-774d5f680a1f-kube-api-access-46c8m" pod "5ce3e833-d075-43d6-b477-774d5f680a1f" (UID: "5ce3e833-d075-43d6-b477-774d5f680a1f"). InnerVolumeSpecName "kube-api-access-46c8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:59.462008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.461987    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-xks-ca" pod "5ce3e833-d075-43d6-b477-774d5f680a1f" (UID: "5ce3e833-d075-43d6-b477-774d5f680a1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:59.561003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.560972    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce3e833-d075-43d6-b477-774d5f680a1f-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:59.561003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.561003    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46c8m\" (UniqueName: \"kubernetes.io/projected/5ce3e833-d075-43d6-b477-774d5f680a1f-kube-api-access-46c8m\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:00:59.744568 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:00:59.744540    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"]
Oct 30 21:00:59.745189 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:00:59.745159    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5871f831_513b_4c57_9be3_2b5b296dbe00.slice/crio-e1d561f6d42501673783d1a5ef710c991bb545dfc765266d21c1df685f41a3f9 WatchSource:0}: Error finding container e1d561f6d42501673783d1a5ef710c991bb545dfc765266d21c1df685f41a3f9: Status 404 returned error can't find the container with id e1d561f6d42501673783d1a5ef710c991bb545dfc765266d21c1df685f41a3f9
Oct 30 21:01:00.216942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:00.210196    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2" event={"ID":"5ce3e833-d075-43d6-b477-774d5f680a1f","Type":"ContainerDied","Data":"ff824ca12957e0958d799b3012bc0ecfa5d760d6720c7226b38cc822687ff958"}
Oct 30 21:01:00.216942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:00.210274    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff824ca12957e0958d799b3012bc0ecfa5d760d6720c7226b38cc822687ff958"
Oct 30 21:01:00.216942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:00.210366    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"
Oct 30 21:01:00.216942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:00.214431    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg" event={"ID":"5871f831-513b-4c57-9be3-2b5b296dbe00","Type":"ContainerStarted","Data":"34fa5da94cea12b47b690bd597b310177a58b7c096acf925e7c53bca83ed249a"}
Oct 30 21:01:00.216942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:00.214500    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg" event={"ID":"5871f831-513b-4c57-9be3-2b5b296dbe00","Type":"ContainerStarted","Data":"e1d561f6d42501673783d1a5ef710c991bb545dfc765266d21c1df685f41a3f9"}
Oct 30 21:01:00.240941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:00.238950    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg" podStartSLOduration=1.238927254 podStartE2EDuration="1.238927254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:00.237045687 +0000 UTC m=+1426.728862230" watchObservedRunningTime="2026-10-30 21:01:00.238927254 +0000 UTC m=+1426.730743798"
Oct 30 21:01:01.223400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:01.223362    4810 generic.go:356] "Generic (PLEG): container finished" podID="5871f831-513b-4c57-9be3-2b5b296dbe00" containerID="34fa5da94cea12b47b690bd597b310177a58b7c096acf925e7c53bca83ed249a" exitCode=0
Oct 30 21:01:01.223814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:01.223443    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg" event={"ID":"5871f831-513b-4c57-9be3-2b5b296dbe00","Type":"ContainerDied","Data":"34fa5da94cea12b47b690bd597b310177a58b7c096acf925e7c53bca83ed249a"}
Oct 30 21:01:02.403015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:02.402988    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:01:02.521895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:02.521800    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-xks-ca\") pod \"5871f831-513b-4c57-9be3-2b5b296dbe00\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") "
Oct 30 21:01:02.521895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:02.521847    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-openshift-ca\") pod \"5871f831-513b-4c57-9be3-2b5b296dbe00\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") "
Oct 30 21:01:02.522122 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:02.521981    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5871f831-513b-4c57-9be3-2b5b296dbe00-kube-api-access-wwjgg\" (UniqueName: \"kubernetes.io/projected/5871f831-513b-4c57-9be3-2b5b296dbe00-kube-api-access-wwjgg\") pod \"5871f831-513b-4c57-9be3-2b5b296dbe00\" (UID: \"5871f831-513b-4c57-9be3-2b5b296dbe00\") "
Oct 30 21:01:02.522341 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:02.522308    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-openshift-ca" pod "5871f831-513b-4c57-9be3-2b5b296dbe00" (UID: "5871f831-513b-4c57-9be3-2b5b296dbe00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:02.524026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:02.524000    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-xks-ca" pod "5871f831-513b-4c57-9be3-2b5b296dbe00" (UID: "5871f831-513b-4c57-9be3-2b5b296dbe00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:02.524164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:02.524143    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5871f831-513b-4c57-9be3-2b5b296dbe00-kube-api-access-wwjgg" pod "5871f831-513b-4c57-9be3-2b5b296dbe00" (UID: "5871f831-513b-4c57-9be3-2b5b296dbe00"). InnerVolumeSpecName "kube-api-access-wwjgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:02.623920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:02.623879    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwjgg\" (UniqueName: \"kubernetes.io/projected/5871f831-513b-4c57-9be3-2b5b296dbe00-kube-api-access-wwjgg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:02.623920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:02.623913    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:02.623920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:02.623925    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5871f831-513b-4c57-9be3-2b5b296dbe00-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:03.240293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:03.240261    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"
Oct 30 21:01:03.240463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:03.240290    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg" event={"ID":"5871f831-513b-4c57-9be3-2b5b296dbe00","Type":"ContainerDied","Data":"e1d561f6d42501673783d1a5ef710c991bb545dfc765266d21c1df685f41a3f9"}
Oct 30 21:01:03.240463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:03.240335    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1d561f6d42501673783d1a5ef710c991bb545dfc765266d21c1df685f41a3f9"
Oct 30 21:01:10.562037 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:10.561998    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"]
Oct 30 21:01:10.565300 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:10.565274    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b80638-q62sh"]
Oct 30 21:01:11.663576 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.663541    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8a98c9-ab84-4b93-b7f8-8572784ed69f" path="/var/lib/kubelet/pods/6f8a98c9-ab84-4b93-b7f8-8572784ed69f/volumes"
Oct 30 21:01:11.727801 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.727745    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"]
Oct 30 21:01:11.728477 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.728461    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5ce3e833-d075-43d6-b477-774d5f680a1f" containerName="revoke-keys"
Oct 30 21:01:11.728523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.728481    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce3e833-d075-43d6-b477-774d5f680a1f" containerName="revoke-keys"
Oct 30 21:01:11.728523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.728491    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5871f831-513b-4c57-9be3-2b5b296dbe00" containerName="revoke-keys"
Oct 30 21:01:11.728523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.728496    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5871f831-513b-4c57-9be3-2b5b296dbe00" containerName="revoke-keys"
Oct 30 21:01:11.728621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.728605    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce3e833-d075-43d6-b477-774d5f680a1f" containerName="revoke-keys"
Oct 30 21:01:11.728621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.728617    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5871f831-513b-4c57-9be3-2b5b296dbe00" containerName="revoke-keys"
Oct 30 21:01:11.733680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.733662    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:11.741020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.740990    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"]
Oct 30 21:01:11.741771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.741743    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:11.745569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.745334    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:01:11.858097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.858059    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wq6c\" (UniqueName: \"kubernetes.io/projected/a971368c-d08f-42c3-a2e9-8ece37f71254-kube-api-access-6wq6c\") pod \"maas-api-revoke-sub-81d8571b-zjg5w\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:11.858097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.858109    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81d8571b-zjg5w\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:11.858346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.858157    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81d8571b-zjg5w\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:11.959022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.958931    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wq6c\" (UniqueName: \"kubernetes.io/projected/a971368c-d08f-42c3-a2e9-8ece37f71254-kube-api-access-6wq6c\") pod \"maas-api-revoke-sub-81d8571b-zjg5w\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:11.959022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.958980    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81d8571b-zjg5w\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:11.959022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.959015    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81d8571b-zjg5w\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:11.959675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.959650    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81d8571b-zjg5w\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:11.961381 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.961356    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81d8571b-zjg5w\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:11.967550 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:11.967529    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wq6c\" (UniqueName: \"kubernetes.io/projected/a971368c-d08f-42c3-a2e9-8ece37f71254-kube-api-access-6wq6c\") pod \"maas-api-revoke-sub-81d8571b-zjg5w\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:12.059135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.059103    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:12.342632 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:01:12.342594    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda971368c_d08f_42c3_a2e9_8ece37f71254.slice/crio-9aa3266aa8b97c168a2750e69088ae1a3a423a17f0320c9a821d35b7c3cbeb82 WatchSource:0}: Error finding container 9aa3266aa8b97c168a2750e69088ae1a3a423a17f0320c9a821d35b7c3cbeb82: Status 404 returned error can't find the container with id 9aa3266aa8b97c168a2750e69088ae1a3a423a17f0320c9a821d35b7c3cbeb82
Oct 30 21:01:12.343668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.343642    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"]
Oct 30 21:01:12.596971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.596943    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"]
Oct 30 21:01:12.602661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.602641    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:12.616217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.616186    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"]
Oct 30 21:01:12.673991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.673962    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5550ae3-4qptm\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:12.674368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.674051    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5550ae3-4qptm\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:12.674368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.674114    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxk9q\" (UniqueName: \"kubernetes.io/projected/4a4aa1d7-fb3f-4739-b215-ef573d30d152-kube-api-access-nxk9q\") pod \"maas-api-revoke-sub-c5550ae3-4qptm\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:12.775577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.775548    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5550ae3-4qptm\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:12.775749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.775690    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5550ae3-4qptm\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:12.775812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.775771    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxk9q\" (UniqueName: \"kubernetes.io/projected/4a4aa1d7-fb3f-4739-b215-ef573d30d152-kube-api-access-nxk9q\") pod \"maas-api-revoke-sub-c5550ae3-4qptm\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:12.776360 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.776338    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5550ae3-4qptm\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:12.778101 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.778083    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5550ae3-4qptm\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:12.789900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.789876    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxk9q\" (UniqueName: \"kubernetes.io/projected/4a4aa1d7-fb3f-4739-b215-ef573d30d152-kube-api-access-nxk9q\") pod \"maas-api-revoke-sub-c5550ae3-4qptm\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:12.921630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:12.921596    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:13.206345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:13.206307    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"]
Oct 30 21:01:13.206870 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:01:13.206823    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4aa1d7_fb3f_4739_b215_ef573d30d152.slice/crio-b9335ecadb88b9554ae4fbfc3377b914b593b5ebfde1b91dd87caed52bff460c WatchSource:0}: Error finding container b9335ecadb88b9554ae4fbfc3377b914b593b5ebfde1b91dd87caed52bff460c: Status 404 returned error can't find the container with id b9335ecadb88b9554ae4fbfc3377b914b593b5ebfde1b91dd87caed52bff460c
Oct 30 21:01:13.346334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:13.346296    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm" event={"ID":"4a4aa1d7-fb3f-4739-b215-ef573d30d152","Type":"ContainerStarted","Data":"b9335ecadb88b9554ae4fbfc3377b914b593b5ebfde1b91dd87caed52bff460c"}
Oct 30 21:01:13.348040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:13.348015    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w" event={"ID":"a971368c-d08f-42c3-a2e9-8ece37f71254","Type":"ContainerStarted","Data":"36417ba24c9f6e029f3e49c5860678d8ab2e4b3e1cdd46327579d2a9bcb42464"}
Oct 30 21:01:13.348177 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:13.348061    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w" event={"ID":"a971368c-d08f-42c3-a2e9-8ece37f71254","Type":"ContainerStarted","Data":"9aa3266aa8b97c168a2750e69088ae1a3a423a17f0320c9a821d35b7c3cbeb82"}
Oct 30 21:01:13.368800 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:13.368753    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w" podStartSLOduration=2.368740823 podStartE2EDuration="2.368740823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:13.366556184 +0000 UTC m=+1439.858372728" watchObservedRunningTime="2026-10-30 21:01:13.368740823 +0000 UTC m=+1439.860557364"
Oct 30 21:01:14.359133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:14.359049    4810 generic.go:356] "Generic (PLEG): container finished" podID="a971368c-d08f-42c3-a2e9-8ece37f71254" containerID="36417ba24c9f6e029f3e49c5860678d8ab2e4b3e1cdd46327579d2a9bcb42464" exitCode=0
Oct 30 21:01:14.359574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:14.359135    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w" event={"ID":"a971368c-d08f-42c3-a2e9-8ece37f71254","Type":"ContainerDied","Data":"36417ba24c9f6e029f3e49c5860678d8ab2e4b3e1cdd46327579d2a9bcb42464"}
Oct 30 21:01:14.360967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:14.360943    4810 generic.go:356] "Generic (PLEG): container finished" podID="4a4aa1d7-fb3f-4739-b215-ef573d30d152" containerID="7c9aaef4b7926e2679b94b3827ddc7fd8d64d1dfdf90876b61f25227fdaaff6c" exitCode=0
Oct 30 21:01:14.361071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:14.361024    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm" event={"ID":"4a4aa1d7-fb3f-4739-b215-ef573d30d152","Type":"ContainerDied","Data":"7c9aaef4b7926e2679b94b3827ddc7fd8d64d1dfdf90876b61f25227fdaaff6c"}
Oct 30 21:01:15.137695 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.137663    4810 scope.go:122] "RemoveContainer" containerID="69ced85f2764d73ed2142bc6a3d404909d4a83579f790febba277c9aa339d345"
Oct 30 21:01:15.148197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.148177    4810 scope.go:122] "RemoveContainer" containerID="a4262b7d9c07c5213f29ca6c36eaa2fa35aab99efe8474693a96aa48babe0b7d"
Oct 30 21:01:15.157256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.157236    4810 scope.go:122] "RemoveContainer" containerID="dd53a81c5ed927d44f9b6eb308ab69d6f8b846b880be07e14df999102642c10e"
Oct 30 21:01:15.165788 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.165773    4810 scope.go:122] "RemoveContainer" containerID="9cf1a65535916ae54df5294f6c14e47a5ed2ee2334023deac3acbaac5b250859"
Oct 30 21:01:15.174443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.174412    4810 scope.go:122] "RemoveContainer" containerID="1b156f84fe8dc4b209ab4990b805f8dce5294d73459534dde6522998876b372f"
Oct 30 21:01:15.569004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.568984    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:15.571003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.570982    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:15.635715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.635677    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-xks-ca\") pod \"a971368c-d08f-42c3-a2e9-8ece37f71254\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") "
Oct 30 21:01:15.635930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.635740    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4aa1d7-fb3f-4739-b215-ef573d30d152-kube-api-access-nxk9q\" (UniqueName: \"kubernetes.io/projected/4a4aa1d7-fb3f-4739-b215-ef573d30d152-kube-api-access-nxk9q\") pod \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") "
Oct 30 21:01:15.636011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.635946    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-openshift-ca\") pod \"a971368c-d08f-42c3-a2e9-8ece37f71254\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") "
Oct 30 21:01:15.636353 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.636325    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-openshift-ca" pod "a971368c-d08f-42c3-a2e9-8ece37f71254" (UID: "a971368c-d08f-42c3-a2e9-8ece37f71254"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:15.636454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.636436    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-openshift-ca\") pod \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") "
Oct 30 21:01:15.636580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.636559    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-xks-ca\") pod \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\" (UID: \"4a4aa1d7-fb3f-4739-b215-ef573d30d152\") "
Oct 30 21:01:15.636716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.636612    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a971368c-d08f-42c3-a2e9-8ece37f71254-kube-api-access-6wq6c\" (UniqueName: \"kubernetes.io/projected/a971368c-d08f-42c3-a2e9-8ece37f71254-kube-api-access-6wq6c\") pod \"a971368c-d08f-42c3-a2e9-8ece37f71254\" (UID: \"a971368c-d08f-42c3-a2e9-8ece37f71254\") "
Oct 30 21:01:15.637238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.637211    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-openshift-ca" pod "4a4aa1d7-fb3f-4739-b215-ef573d30d152" (UID: "4a4aa1d7-fb3f-4739-b215-ef573d30d152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:15.637558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.637534    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:15.637677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.637662    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:15.638354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.638322    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-xks-ca" pod "a971368c-d08f-42c3-a2e9-8ece37f71254" (UID: "a971368c-d08f-42c3-a2e9-8ece37f71254"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:15.639575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.639549    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-xks-ca" pod "4a4aa1d7-fb3f-4739-b215-ef573d30d152" (UID: "4a4aa1d7-fb3f-4739-b215-ef573d30d152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:15.642348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.642315    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a971368c-d08f-42c3-a2e9-8ece37f71254-kube-api-access-6wq6c" pod "a971368c-d08f-42c3-a2e9-8ece37f71254" (UID: "a971368c-d08f-42c3-a2e9-8ece37f71254"). InnerVolumeSpecName "kube-api-access-6wq6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:15.642432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.642381    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4aa1d7-fb3f-4739-b215-ef573d30d152-kube-api-access-nxk9q" pod "4a4aa1d7-fb3f-4739-b215-ef573d30d152" (UID: "4a4aa1d7-fb3f-4739-b215-ef573d30d152"). InnerVolumeSpecName "kube-api-access-nxk9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:15.739724 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.739692    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a4aa1d7-fb3f-4739-b215-ef573d30d152-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:15.739890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.739728    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wq6c\" (UniqueName: \"kubernetes.io/projected/a971368c-d08f-42c3-a2e9-8ece37f71254-kube-api-access-6wq6c\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:15.739890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.739753    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971368c-d08f-42c3-a2e9-8ece37f71254-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:15.739890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:15.739771    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxk9q\" (UniqueName: \"kubernetes.io/projected/4a4aa1d7-fb3f-4739-b215-ef573d30d152-kube-api-access-nxk9q\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:16.377373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:16.377341    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"
Oct 30 21:01:16.377581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:16.377378    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w" event={"ID":"a971368c-d08f-42c3-a2e9-8ece37f71254","Type":"ContainerDied","Data":"9aa3266aa8b97c168a2750e69088ae1a3a423a17f0320c9a821d35b7c3cbeb82"}
Oct 30 21:01:16.377581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:16.377427    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aa3266aa8b97c168a2750e69088ae1a3a423a17f0320c9a821d35b7c3cbeb82"
Oct 30 21:01:16.378947 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:16.378917    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm" event={"ID":"4a4aa1d7-fb3f-4739-b215-ef573d30d152","Type":"ContainerDied","Data":"b9335ecadb88b9554ae4fbfc3377b914b593b5ebfde1b91dd87caed52bff460c"}
Oct 30 21:01:16.378947 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:16.378935    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"
Oct 30 21:01:16.379143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:16.378965    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9335ecadb88b9554ae4fbfc3377b914b593b5ebfde1b91dd87caed52bff460c"
Oct 30 21:01:17.662900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:17.662873    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:01:20.569882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:20.565532    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 30 21:01:20.633880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:20.629013    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:20.657120 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:20.657070    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:20.746991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:20.746939    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:21.775188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:21.772249    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:21.775188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:21.775175    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 30 21:01:21.775893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:21.775607    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 21:01:21.775893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:21.775665    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 21:01:22.170538 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:22.147272    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:01:22.170538 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:22.147353    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:22.299970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:22.299926    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 21:01:22.607915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:22.607804    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 21:01:24.323608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.323573    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"]
Oct 30 21:01:24.324276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.324260    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a971368c-d08f-42c3-a2e9-8ece37f71254" containerName="revoke-keys"
Oct 30 21:01:24.324322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.324277    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a971368c-d08f-42c3-a2e9-8ece37f71254" containerName="revoke-keys"
Oct 30 21:01:24.324358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.324322    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4a4aa1d7-fb3f-4739-b215-ef573d30d152" containerName="revoke-keys"
Oct 30 21:01:24.324358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.324328    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4aa1d7-fb3f-4739-b215-ef573d30d152" containerName="revoke-keys"
Oct 30 21:01:24.324444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.324434    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a971368c-d08f-42c3-a2e9-8ece37f71254" containerName="revoke-keys"
Oct 30 21:01:24.324481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.324455    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4aa1d7-fb3f-4739-b215-ef573d30d152" containerName="revoke-keys"
Oct 30 21:01:24.329417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.329400    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.333782 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.333757    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:24.333970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.333953    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:01:24.338315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.338073    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"]
Oct 30 21:01:24.448911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.448845    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42e81d5-h54q8\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.450062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.450040    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tct\" (UniqueName: \"kubernetes.io/projected/30cd1a7b-7650-4a0c-8b04-a11fd718032c-kube-api-access-v8tct\") pod \"maas-api-revoke-sub-b42e81d5-h54q8\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.450280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.450263    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42e81d5-h54q8\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.551147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.551117    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tct\" (UniqueName: \"kubernetes.io/projected/30cd1a7b-7650-4a0c-8b04-a11fd718032c-kube-api-access-v8tct\") pod \"maas-api-revoke-sub-b42e81d5-h54q8\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.551332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.551248    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42e81d5-h54q8\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.551408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.551365    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42e81d5-h54q8\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.552132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.552100    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42e81d5-h54q8\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.553846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.553819    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42e81d5-h54q8\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.560237 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.560210    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tct\" (UniqueName: \"kubernetes.io/projected/30cd1a7b-7650-4a0c-8b04-a11fd718032c-kube-api-access-v8tct\") pod \"maas-api-revoke-sub-b42e81d5-h54q8\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.650008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.649973    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:24.974918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:24.974888    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"]
Oct 30 21:01:24.977152 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:01:24.977112    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cd1a7b_7650_4a0c_8b04_a11fd718032c.slice/crio-b798dabaf7ec8015a9d5a7734757edd4a64ef3f27bd4b07e1c6df207d82e5dab WatchSource:0}: Error finding container b798dabaf7ec8015a9d5a7734757edd4a64ef3f27bd4b07e1c6df207d82e5dab: Status 404 returned error can't find the container with id b798dabaf7ec8015a9d5a7734757edd4a64ef3f27bd4b07e1c6df207d82e5dab
Oct 30 21:01:25.155619 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:01:25.155584    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:01:25.910503 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:25.910472    4810 generic.go:356] "Generic (PLEG): container finished" podID="30cd1a7b-7650-4a0c-8b04-a11fd718032c" containerID="6524f9a9c819ca151e7638677faab922d7a1e0fdda1e602c13e84610e518d247" exitCode=0
Oct 30 21:01:25.910938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:25.910515    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8" event={"ID":"30cd1a7b-7650-4a0c-8b04-a11fd718032c","Type":"ContainerDied","Data":"6524f9a9c819ca151e7638677faab922d7a1e0fdda1e602c13e84610e518d247"}
Oct 30 21:01:25.910938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:25.910564    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8" event={"ID":"30cd1a7b-7650-4a0c-8b04-a11fd718032c","Type":"ContainerStarted","Data":"b798dabaf7ec8015a9d5a7734757edd4a64ef3f27bd4b07e1c6df207d82e5dab"}
Oct 30 21:01:27.088292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.088267    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:27.185966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.185874    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30cd1a7b-7650-4a0c-8b04-a11fd718032c-kube-api-access-v8tct\" (UniqueName: \"kubernetes.io/projected/30cd1a7b-7650-4a0c-8b04-a11fd718032c-kube-api-access-v8tct\") pod \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") "
Oct 30 21:01:27.185966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.185935    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-xks-ca\") pod \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") "
Oct 30 21:01:27.186213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.186183    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-openshift-ca\") pod \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\" (UID: \"30cd1a7b-7650-4a0c-8b04-a11fd718032c\") "
Oct 30 21:01:27.186622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.186577    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-openshift-ca" pod "30cd1a7b-7650-4a0c-8b04-a11fd718032c" (UID: "30cd1a7b-7650-4a0c-8b04-a11fd718032c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:27.186768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.186684    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:27.188053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.188020    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-xks-ca" pod "30cd1a7b-7650-4a0c-8b04-a11fd718032c" (UID: "30cd1a7b-7650-4a0c-8b04-a11fd718032c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:27.188468 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.188442    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cd1a7b-7650-4a0c-8b04-a11fd718032c-kube-api-access-v8tct" pod "30cd1a7b-7650-4a0c-8b04-a11fd718032c" (UID: "30cd1a7b-7650-4a0c-8b04-a11fd718032c"). InnerVolumeSpecName "kube-api-access-v8tct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:27.288055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.288020    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8tct\" (UniqueName: \"kubernetes.io/projected/30cd1a7b-7650-4a0c-8b04-a11fd718032c-kube-api-access-v8tct\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:27.288055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.288051    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cd1a7b-7650-4a0c-8b04-a11fd718032c-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:27.928759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.928684    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"
Oct 30 21:01:27.928759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.928713    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8" event={"ID":"30cd1a7b-7650-4a0c-8b04-a11fd718032c","Type":"ContainerDied","Data":"b798dabaf7ec8015a9d5a7734757edd4a64ef3f27bd4b07e1c6df207d82e5dab"}
Oct 30 21:01:27.928992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:27.928773    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b798dabaf7ec8015a9d5a7734757edd4a64ef3f27bd4b07e1c6df207d82e5dab"
Oct 30 21:01:28.329040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.328922    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"]
Oct 30 21:01:28.329600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.329581    4810 cpu_manager.go:414] "containerMap: removing container" podUID="30cd1a7b-7650-4a0c-8b04-a11fd718032c" containerName="revoke-keys"
Oct 30 21:01:28.329665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.329602    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30cd1a7b-7650-4a0c-8b04-a11fd718032c" containerName="revoke-keys"
Oct 30 21:01:28.329767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.329749    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="30cd1a7b-7650-4a0c-8b04-a11fd718032c" containerName="revoke-keys"
Oct 30 21:01:28.334046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.334007    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"]
Oct 30 21:01:28.334207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.334188    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.344850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.344627    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:01:28.345251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.345227    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:28.418258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.418221    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-411319da-ggkrs\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.418425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.418288    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-411319da-ggkrs\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.418425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.418380    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4hr\" (UniqueName: \"kubernetes.io/projected/fc6bb593-9e72-4a60-b93b-00649dfaf912-kube-api-access-6c4hr\") pod \"maas-api-revoke-sub-411319da-ggkrs\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.520009 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.519969    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-411319da-ggkrs\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.520203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.520046    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-411319da-ggkrs\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.520203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.520077    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4hr\" (UniqueName: \"kubernetes.io/projected/fc6bb593-9e72-4a60-b93b-00649dfaf912-kube-api-access-6c4hr\") pod \"maas-api-revoke-sub-411319da-ggkrs\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.520771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.520743    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-411319da-ggkrs\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.522433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.522415    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-411319da-ggkrs\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.528484 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.528459    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4hr\" (UniqueName: \"kubernetes.io/projected/fc6bb593-9e72-4a60-b93b-00649dfaf912-kube-api-access-6c4hr\") pod \"maas-api-revoke-sub-411319da-ggkrs\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.660963 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.660932    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:28.964888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:28.964725    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"]
Oct 30 21:01:28.966313 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:01:28.966283    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6bb593_9e72_4a60_b93b_00649dfaf912.slice/crio-3be9610a704fe1643e14bee067869dad6557fbe9e4d1f41519ffdfe53678dc56 WatchSource:0}: Error finding container 3be9610a704fe1643e14bee067869dad6557fbe9e4d1f41519ffdfe53678dc56: Status 404 returned error can't find the container with id 3be9610a704fe1643e14bee067869dad6557fbe9e4d1f41519ffdfe53678dc56
Oct 30 21:01:29.953040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:29.952993    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs" event={"ID":"fc6bb593-9e72-4a60-b93b-00649dfaf912","Type":"ContainerStarted","Data":"807e6b0dc413ee0f4b5bf46bcf10c25897faea71bd5ac6d778a291364fd15dbc"}
Oct 30 21:01:29.953600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:29.953077    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs" event={"ID":"fc6bb593-9e72-4a60-b93b-00649dfaf912","Type":"ContainerStarted","Data":"3be9610a704fe1643e14bee067869dad6557fbe9e4d1f41519ffdfe53678dc56"}
Oct 30 21:01:29.972461 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:29.972398    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs" podStartSLOduration=1.972378765 podStartE2EDuration="1.972378765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:29.97015499 +0000 UTC m=+1456.461971537" watchObservedRunningTime="2026-10-30 21:01:29.972378765 +0000 UTC m=+1456.464195319"
Oct 30 21:01:30.964435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:30.964391    4810 generic.go:356] "Generic (PLEG): container finished" podID="fc6bb593-9e72-4a60-b93b-00649dfaf912" containerID="807e6b0dc413ee0f4b5bf46bcf10c25897faea71bd5ac6d778a291364fd15dbc" exitCode=0
Oct 30 21:01:30.964813 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:30.964472    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs" event={"ID":"fc6bb593-9e72-4a60-b93b-00649dfaf912","Type":"ContainerDied","Data":"807e6b0dc413ee0f4b5bf46bcf10c25897faea71bd5ac6d778a291364fd15dbc"}
Oct 30 21:01:32.057918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.055296    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"]
Oct 30 21:01:32.070222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.062998    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"]
Oct 30 21:01:32.070222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.063116    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.155359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.155330    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:32.174831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.174782    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m77rp\" (UniqueName: \"kubernetes.io/projected/a541be3f-37eb-4ca3-970a-7032c68188dc-kube-api-access-m77rp\") pod \"maas-api-revoke-sub-b5d0fd46-7z8q2\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.175050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.174953    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d0fd46-7z8q2\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.175121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.175065    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d0fd46-7z8q2\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.276811 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.276726    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc6bb593-9e72-4a60-b93b-00649dfaf912-kube-api-access-6c4hr\" (UniqueName: \"kubernetes.io/projected/fc6bb593-9e72-4a60-b93b-00649dfaf912-kube-api-access-6c4hr\") pod \"fc6bb593-9e72-4a60-b93b-00649dfaf912\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") "
Oct 30 21:01:32.276811 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.276769    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-xks-ca\") pod \"fc6bb593-9e72-4a60-b93b-00649dfaf912\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") "
Oct 30 21:01:32.276811 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.276798    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-openshift-ca\") pod \"fc6bb593-9e72-4a60-b93b-00649dfaf912\" (UID: \"fc6bb593-9e72-4a60-b93b-00649dfaf912\") "
Oct 30 21:01:32.277135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.277045    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d0fd46-7z8q2\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.277203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.277187    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m77rp\" (UniqueName: \"kubernetes.io/projected/a541be3f-37eb-4ca3-970a-7032c68188dc-kube-api-access-m77rp\") pod \"maas-api-revoke-sub-b5d0fd46-7z8q2\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.277279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.277250    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-openshift-ca" pod "fc6bb593-9e72-4a60-b93b-00649dfaf912" (UID: "fc6bb593-9e72-4a60-b93b-00649dfaf912"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:32.277364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.277347    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d0fd46-7z8q2\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.277548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.277516    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:32.277917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.277893    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d0fd46-7z8q2\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.279529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.279477    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-xks-ca" pod "fc6bb593-9e72-4a60-b93b-00649dfaf912" (UID: "fc6bb593-9e72-4a60-b93b-00649dfaf912"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:32.279529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.279508    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6bb593-9e72-4a60-b93b-00649dfaf912-kube-api-access-6c4hr" pod "fc6bb593-9e72-4a60-b93b-00649dfaf912" (UID: "fc6bb593-9e72-4a60-b93b-00649dfaf912"). InnerVolumeSpecName "kube-api-access-6c4hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:32.279875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.279844    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d0fd46-7z8q2\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.286306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.286272    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m77rp\" (UniqueName: \"kubernetes.io/projected/a541be3f-37eb-4ca3-970a-7032c68188dc-kube-api-access-m77rp\") pod \"maas-api-revoke-sub-b5d0fd46-7z8q2\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.378700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.378666    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c4hr\" (UniqueName: \"kubernetes.io/projected/fc6bb593-9e72-4a60-b93b-00649dfaf912-kube-api-access-6c4hr\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:32.378700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.378696    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc6bb593-9e72-4a60-b93b-00649dfaf912-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:32.401952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.401918    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:32.686387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.686357    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"]
Oct 30 21:01:32.687280 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:01:32.687244    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda541be3f_37eb_4ca3_970a_7032c68188dc.slice/crio-beed8d75c8908baee565b920a9506131dedbf303eb50365bfcff254cc2816921 WatchSource:0}: Error finding container beed8d75c8908baee565b920a9506131dedbf303eb50365bfcff254cc2816921: Status 404 returned error can't find the container with id beed8d75c8908baee565b920a9506131dedbf303eb50365bfcff254cc2816921
Oct 30 21:01:32.984007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.983975    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2" event={"ID":"a541be3f-37eb-4ca3-970a-7032c68188dc","Type":"ContainerStarted","Data":"beed8d75c8908baee565b920a9506131dedbf303eb50365bfcff254cc2816921"}
Oct 30 21:01:32.985818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.985796    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"
Oct 30 21:01:32.985971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.985797    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs" event={"ID":"fc6bb593-9e72-4a60-b93b-00649dfaf912","Type":"ContainerDied","Data":"3be9610a704fe1643e14bee067869dad6557fbe9e4d1f41519ffdfe53678dc56"}
Oct 30 21:01:32.985971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:32.985915    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3be9610a704fe1643e14bee067869dad6557fbe9e4d1f41519ffdfe53678dc56"
Oct 30 21:01:33.994814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:33.994774    4810 generic.go:356] "Generic (PLEG): container finished" podID="a541be3f-37eb-4ca3-970a-7032c68188dc" containerID="fc1314367c9ea2c0f291f7a0f1c1d2be1df3450f3cbf47dd9292679099bf027a" exitCode=0
Oct 30 21:01:33.995321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:33.994850    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2" event={"ID":"a541be3f-37eb-4ca3-970a-7032c68188dc","Type":"ContainerDied","Data":"fc1314367c9ea2c0f291f7a0f1c1d2be1df3450f3cbf47dd9292679099bf027a"}
Oct 30 21:01:35.167334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:35.167307    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:35.323257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:35.323176    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-openshift-ca\") pod \"a541be3f-37eb-4ca3-970a-7032c68188dc\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") "
Oct 30 21:01:35.323421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:35.323307    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a541be3f-37eb-4ca3-970a-7032c68188dc-kube-api-access-m77rp\" (UniqueName: \"kubernetes.io/projected/a541be3f-37eb-4ca3-970a-7032c68188dc-kube-api-access-m77rp\") pod \"a541be3f-37eb-4ca3-970a-7032c68188dc\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") "
Oct 30 21:01:35.323486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:35.323467    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-xks-ca\") pod \"a541be3f-37eb-4ca3-970a-7032c68188dc\" (UID: \"a541be3f-37eb-4ca3-970a-7032c68188dc\") "
Oct 30 21:01:35.323610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:35.323590    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-openshift-ca" pod "a541be3f-37eb-4ca3-970a-7032c68188dc" (UID: "a541be3f-37eb-4ca3-970a-7032c68188dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:35.323929 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:35.323905    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.325397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:35.325378    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-xks-ca" pod "a541be3f-37eb-4ca3-970a-7032c68188dc" (UID: "a541be3f-37eb-4ca3-970a-7032c68188dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:35.325495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:35.325479    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a541be3f-37eb-4ca3-970a-7032c68188dc-kube-api-access-m77rp" pod "a541be3f-37eb-4ca3-970a-7032c68188dc" (UID: "a541be3f-37eb-4ca3-970a-7032c68188dc"). InnerVolumeSpecName "kube-api-access-m77rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:35.425146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:35.425104    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m77rp\" (UniqueName: \"kubernetes.io/projected/a541be3f-37eb-4ca3-970a-7032c68188dc-kube-api-access-m77rp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.425146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:35.425140    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a541be3f-37eb-4ca3-970a-7032c68188dc-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:36.017021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.016934    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"
Oct 30 21:01:36.017187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.016937    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2" event={"ID":"a541be3f-37eb-4ca3-970a-7032c68188dc","Type":"ContainerDied","Data":"beed8d75c8908baee565b920a9506131dedbf303eb50365bfcff254cc2816921"}
Oct 30 21:01:36.017187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.017082    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beed8d75c8908baee565b920a9506131dedbf303eb50365bfcff254cc2816921"
Oct 30 21:01:36.441362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.441331    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"]
Oct 30 21:01:36.441933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.441919    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fc6bb593-9e72-4a60-b93b-00649dfaf912" containerName="revoke-keys"
Oct 30 21:01:36.441933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.441933    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6bb593-9e72-4a60-b93b-00649dfaf912" containerName="revoke-keys"
Oct 30 21:01:36.442021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.441945    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a541be3f-37eb-4ca3-970a-7032c68188dc" containerName="revoke-keys"
Oct 30 21:01:36.442021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.441950    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a541be3f-37eb-4ca3-970a-7032c68188dc" containerName="revoke-keys"
Oct 30 21:01:36.442084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.442062    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6bb593-9e72-4a60-b93b-00649dfaf912" containerName="revoke-keys"
Oct 30 21:01:36.442084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.442071    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a541be3f-37eb-4ca3-970a-7032c68188dc" containerName="revoke-keys"
Oct 30 21:01:36.445558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.445542    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:36.449788 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.449719    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:01:36.449788 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.449711    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:36.455632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.455602    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"]
Oct 30 21:01:36.546242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.546204    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-169cbb63-t8pzv\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:36.546453 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.546430    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwnk5\" (UniqueName: \"kubernetes.io/projected/17ce23a0-3974-4c00-b4a7-b0920d00748d-kube-api-access-gwnk5\") pod \"maas-api-revoke-sub-169cbb63-t8pzv\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:36.546519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.546497    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-169cbb63-t8pzv\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:36.648059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.648021    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwnk5\" (UniqueName: \"kubernetes.io/projected/17ce23a0-3974-4c00-b4a7-b0920d00748d-kube-api-access-gwnk5\") pod \"maas-api-revoke-sub-169cbb63-t8pzv\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:36.648281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.648095    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-169cbb63-t8pzv\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:36.648281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.648151    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-169cbb63-t8pzv\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:36.649061 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.649029    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-169cbb63-t8pzv\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:36.657618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.657573    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-169cbb63-t8pzv\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:36.657810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.657723    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwnk5\" (UniqueName: \"kubernetes.io/projected/17ce23a0-3974-4c00-b4a7-b0920d00748d-kube-api-access-gwnk5\") pod \"maas-api-revoke-sub-169cbb63-t8pzv\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:36.765428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:36.765352    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:37.049591 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:37.049562    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"]
Oct 30 21:01:37.049970 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:01:37.049940    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ce23a0_3974_4c00_b4a7_b0920d00748d.slice/crio-1bf96abbd6ce14cf9ec7547d3b5fd16520d60d226174527a35c3f9ec3f7ef004 WatchSource:0}: Error finding container 1bf96abbd6ce14cf9ec7547d3b5fd16520d60d226174527a35c3f9ec3f7ef004: Status 404 returned error can't find the container with id 1bf96abbd6ce14cf9ec7547d3b5fd16520d60d226174527a35c3f9ec3f7ef004
Oct 30 21:01:38.035257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:38.035221    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv" event={"ID":"17ce23a0-3974-4c00-b4a7-b0920d00748d","Type":"ContainerStarted","Data":"163b1137ee020ea0d299c420857e5a7e7dfd7d3d38e9a1a85e79da628fcf9619"}
Oct 30 21:01:38.035699 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:38.035282    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv" event={"ID":"17ce23a0-3974-4c00-b4a7-b0920d00748d","Type":"ContainerStarted","Data":"1bf96abbd6ce14cf9ec7547d3b5fd16520d60d226174527a35c3f9ec3f7ef004"}
Oct 30 21:01:38.055527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:38.055480    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv" podStartSLOduration=2.055468282 podStartE2EDuration="2.055468282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:38.0517988 +0000 UTC m=+1464.543615344" watchObservedRunningTime="2026-10-30 21:01:38.055468282 +0000 UTC m=+1464.547284823"
Oct 30 21:01:39.046183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:39.046150    4810 generic.go:356] "Generic (PLEG): container finished" podID="17ce23a0-3974-4c00-b4a7-b0920d00748d" containerID="163b1137ee020ea0d299c420857e5a7e7dfd7d3d38e9a1a85e79da628fcf9619" exitCode=0
Oct 30 21:01:39.046678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:39.046195    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv" event={"ID":"17ce23a0-3974-4c00-b4a7-b0920d00748d","Type":"ContainerDied","Data":"163b1137ee020ea0d299c420857e5a7e7dfd7d3d38e9a1a85e79da628fcf9619"}
Oct 30 21:01:39.184480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:39.184444    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:01:40.621882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:40.616551    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:40.645822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:40.642659    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-xks-ca\") pod \"17ce23a0-3974-4c00-b4a7-b0920d00748d\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") "
Oct 30 21:01:40.645822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:40.642755    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-openshift-ca\") pod \"17ce23a0-3974-4c00-b4a7-b0920d00748d\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") "
Oct 30 21:01:40.645822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:40.642797    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ce23a0-3974-4c00-b4a7-b0920d00748d-kube-api-access-gwnk5\" (UniqueName: \"kubernetes.io/projected/17ce23a0-3974-4c00-b4a7-b0920d00748d-kube-api-access-gwnk5\") pod \"17ce23a0-3974-4c00-b4a7-b0920d00748d\" (UID: \"17ce23a0-3974-4c00-b4a7-b0920d00748d\") "
Oct 30 21:01:40.651290 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:40.650984    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ce23a0-3974-4c00-b4a7-b0920d00748d-kube-api-access-gwnk5" pod "17ce23a0-3974-4c00-b4a7-b0920d00748d" (UID: "17ce23a0-3974-4c00-b4a7-b0920d00748d"). InnerVolumeSpecName "kube-api-access-gwnk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:40.667298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:40.659379    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-openshift-ca" pod "17ce23a0-3974-4c00-b4a7-b0920d00748d" (UID: "17ce23a0-3974-4c00-b4a7-b0920d00748d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:40.683518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:40.683471    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-xks-ca" pod "17ce23a0-3974-4c00-b4a7-b0920d00748d" (UID: "17ce23a0-3974-4c00-b4a7-b0920d00748d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:40.756698 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:40.756606    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:40.756948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:40.756928    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ce23a0-3974-4c00-b4a7-b0920d00748d-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:40.776201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:40.757067    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwnk5\" (UniqueName: \"kubernetes.io/projected/17ce23a0-3974-4c00-b4a7-b0920d00748d-kube-api-access-gwnk5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:41.122933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:41.116294    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv" event={"ID":"17ce23a0-3974-4c00-b4a7-b0920d00748d","Type":"ContainerDied","Data":"1bf96abbd6ce14cf9ec7547d3b5fd16520d60d226174527a35c3f9ec3f7ef004"}
Oct 30 21:01:41.122933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:41.116359    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bf96abbd6ce14cf9ec7547d3b5fd16520d60d226174527a35c3f9ec3f7ef004"
Oct 30 21:01:41.122933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:41.116431    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"
Oct 30 21:01:44.394756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.394723    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"]
Oct 30 21:01:44.395404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.395388    4810 cpu_manager.go:414] "containerMap: removing container" podUID="17ce23a0-3974-4c00-b4a7-b0920d00748d" containerName="revoke-keys"
Oct 30 21:01:44.395469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.395406    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ce23a0-3974-4c00-b4a7-b0920d00748d" containerName="revoke-keys"
Oct 30 21:01:44.395528 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.395499    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ce23a0-3974-4c00-b4a7-b0920d00748d" containerName="revoke-keys"
Oct 30 21:01:44.399491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.399468    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.403808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.403786    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:44.404090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.404072    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:01:44.407437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.407410    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"]
Oct 30 21:01:44.564835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.564802    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ef2303-vzcw4\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.565008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.564938    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ef2303-vzcw4\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.565008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.564968    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhmwv\" (UniqueName: \"kubernetes.io/projected/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-kube-api-access-mhmwv\") pod \"maas-api-revoke-sub-c6ef2303-vzcw4\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.666447 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.666363    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ef2303-vzcw4\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.666447 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.666439    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ef2303-vzcw4\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.666694 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.666461    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhmwv\" (UniqueName: \"kubernetes.io/projected/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-kube-api-access-mhmwv\") pod \"maas-api-revoke-sub-c6ef2303-vzcw4\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.667138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.667117    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ef2303-vzcw4\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.668772 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.668748    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ef2303-vzcw4\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.674746 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.674718    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhmwv\" (UniqueName: \"kubernetes.io/projected/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-kube-api-access-mhmwv\") pod \"maas-api-revoke-sub-c6ef2303-vzcw4\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.721390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.721337    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:44.991031 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.990946    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"]
Oct 30 21:01:44.996604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:44.996576    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.006419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.006325    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"]
Oct 30 21:01:45.023943 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.023914    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"]
Oct 30 21:01:45.026104 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:01:45.026079    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ece89a_a4bb_4bf8_b8b0_4e8655ec5dde.slice/crio-76ed3c7a283cc68cd649ea5848f3da9b8aa831e7c25037fdfc7e24ee6a7e5e42 WatchSource:0}: Error finding container 76ed3c7a283cc68cd649ea5848f3da9b8aa831e7c25037fdfc7e24ee6a7e5e42: Status 404 returned error can't find the container with id 76ed3c7a283cc68cd649ea5848f3da9b8aa831e7c25037fdfc7e24ee6a7e5e42
Oct 30 21:01:45.173171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.173141    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28kdg\" (UniqueName: \"kubernetes.io/projected/b8c6b938-5c3f-4222-89cb-244426dc086e-kube-api-access-28kdg\") pod \"maas-api-revoke-sub-eaeb7edb-7jpb9\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.173350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.173246    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaeb7edb-7jpb9\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.173350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.173304    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaeb7edb-7jpb9\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.209299 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.209255    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4" event={"ID":"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde","Type":"ContainerStarted","Data":"76ed3c7a283cc68cd649ea5848f3da9b8aa831e7c25037fdfc7e24ee6a7e5e42"}
Oct 30 21:01:45.274402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.274326    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28kdg\" (UniqueName: \"kubernetes.io/projected/b8c6b938-5c3f-4222-89cb-244426dc086e-kube-api-access-28kdg\") pod \"maas-api-revoke-sub-eaeb7edb-7jpb9\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.274548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.274428    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaeb7edb-7jpb9\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.274548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.274500    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaeb7edb-7jpb9\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.275336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.275311    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaeb7edb-7jpb9\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.277137 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.277114    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaeb7edb-7jpb9\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.282784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.282764    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28kdg\" (UniqueName: \"kubernetes.io/projected/b8c6b938-5c3f-4222-89cb-244426dc086e-kube-api-access-28kdg\") pod \"maas-api-revoke-sub-eaeb7edb-7jpb9\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.317914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.317847    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:45.608914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:45.608878    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"]
Oct 30 21:01:45.609672 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:01:45.609645    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c6b938_5c3f_4222_89cb_244426dc086e.slice/crio-0b9c8bc6834e05a5482730609a76a4578899e0d930c99888775a6044e58e731a WatchSource:0}: Error finding container 0b9c8bc6834e05a5482730609a76a4578899e0d930c99888775a6044e58e731a: Status 404 returned error can't find the container with id 0b9c8bc6834e05a5482730609a76a4578899e0d930c99888775a6044e58e731a
Oct 30 21:01:46.220001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:46.219912    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9" event={"ID":"b8c6b938-5c3f-4222-89cb-244426dc086e","Type":"ContainerStarted","Data":"d6155ce24af3a289576e2b8a9b812f0431d49d7e0ab4876cf38704ea09282e41"}
Oct 30 21:01:46.220001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:46.219978    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9" event={"ID":"b8c6b938-5c3f-4222-89cb-244426dc086e","Type":"ContainerStarted","Data":"0b9c8bc6834e05a5482730609a76a4578899e0d930c99888775a6044e58e731a"}
Oct 30 21:01:46.221728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:46.221698    4810 generic.go:356] "Generic (PLEG): container finished" podID="f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde" containerID="f05126a1704e9d1000259581e7e0b2a50b93f0765ea1d144b9d39b8018981c68" exitCode=0
Oct 30 21:01:46.221888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:46.221749    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4" event={"ID":"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde","Type":"ContainerDied","Data":"f05126a1704e9d1000259581e7e0b2a50b93f0765ea1d144b9d39b8018981c68"}
Oct 30 21:01:46.236408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:46.236354    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9" podStartSLOduration=2.236337495 podStartE2EDuration="2.236337495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:46.235365099 +0000 UTC m=+1472.727181643" watchObservedRunningTime="2026-10-30 21:01:46.236337495 +0000 UTC m=+1472.728154043"
Oct 30 21:01:47.232451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.232418    4810 generic.go:356] "Generic (PLEG): container finished" podID="b8c6b938-5c3f-4222-89cb-244426dc086e" containerID="d6155ce24af3a289576e2b8a9b812f0431d49d7e0ab4876cf38704ea09282e41" exitCode=0
Oct 30 21:01:47.232842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.232487    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9" event={"ID":"b8c6b938-5c3f-4222-89cb-244426dc086e","Type":"ContainerDied","Data":"d6155ce24af3a289576e2b8a9b812f0431d49d7e0ab4876cf38704ea09282e41"}
Oct 30 21:01:47.413338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.413305    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:47.507098 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.507005    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-xks-ca\") pod \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") "
Oct 30 21:01:47.507098 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.507059    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-openshift-ca\") pod \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") "
Oct 30 21:01:47.507325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.507180    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-kube-api-access-mhmwv\" (UniqueName: \"kubernetes.io/projected/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-kube-api-access-mhmwv\") pod \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\" (UID: \"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde\") "
Oct 30 21:01:47.507583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.507546    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-openshift-ca" pod "f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde" (UID: "f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:47.507795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.507773    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:47.515930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.509449    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-xks-ca" pod "f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde" (UID: "f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:47.515930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.513331    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-kube-api-access-mhmwv" pod "f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde" (UID: "f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde"). InnerVolumeSpecName "kube-api-access-mhmwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:47.609241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.609205    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:47.609241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:47.609239    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhmwv\" (UniqueName: \"kubernetes.io/projected/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde-kube-api-access-mhmwv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:48.241790 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.241759    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"
Oct 30 21:01:48.242220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.241791    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4" event={"ID":"f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde","Type":"ContainerDied","Data":"76ed3c7a283cc68cd649ea5848f3da9b8aa831e7c25037fdfc7e24ee6a7e5e42"}
Oct 30 21:01:48.242220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.241835    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76ed3c7a283cc68cd649ea5848f3da9b8aa831e7c25037fdfc7e24ee6a7e5e42"
Oct 30 21:01:48.459911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.459885    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:48.624820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.624720    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-xks-ca\") pod \"b8c6b938-5c3f-4222-89cb-244426dc086e\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") "
Oct 30 21:01:48.624820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.624820    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-openshift-ca\") pod \"b8c6b938-5c3f-4222-89cb-244426dc086e\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") "
Oct 30 21:01:48.625086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.624893    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c6b938-5c3f-4222-89cb-244426dc086e-kube-api-access-28kdg\" (UniqueName: \"kubernetes.io/projected/b8c6b938-5c3f-4222-89cb-244426dc086e-kube-api-access-28kdg\") pod \"b8c6b938-5c3f-4222-89cb-244426dc086e\" (UID: \"b8c6b938-5c3f-4222-89cb-244426dc086e\") "
Oct 30 21:01:48.625190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.625163    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-openshift-ca" pod "b8c6b938-5c3f-4222-89cb-244426dc086e" (UID: "b8c6b938-5c3f-4222-89cb-244426dc086e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:48.625431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.625416    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:48.627042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.627020    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-xks-ca" pod "b8c6b938-5c3f-4222-89cb-244426dc086e" (UID: "b8c6b938-5c3f-4222-89cb-244426dc086e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:48.627042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.627031    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c6b938-5c3f-4222-89cb-244426dc086e-kube-api-access-28kdg" pod "b8c6b938-5c3f-4222-89cb-244426dc086e" (UID: "b8c6b938-5c3f-4222-89cb-244426dc086e"). InnerVolumeSpecName "kube-api-access-28kdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:48.726519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.726485    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28kdg\" (UniqueName: \"kubernetes.io/projected/b8c6b938-5c3f-4222-89cb-244426dc086e-kube-api-access-28kdg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:48.726519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:48.726514    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c6b938-5c3f-4222-89cb-244426dc086e-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:01:49.250428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:49.250394    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"
Oct 30 21:01:49.250428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:49.250417    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9" event={"ID":"b8c6b938-5c3f-4222-89cb-244426dc086e","Type":"ContainerDied","Data":"0b9c8bc6834e05a5482730609a76a4578899e0d930c99888775a6044e58e731a"}
Oct 30 21:01:49.250889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:49.250475    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9c8bc6834e05a5482730609a76a4578899e0d930c99888775a6044e58e731a"
Oct 30 21:01:55.278008 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:01:55.277603    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:01:55.371260 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.371224    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"]
Oct 30 21:01:55.372507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.372218    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde" containerName="revoke-keys"
Oct 30 21:01:55.372507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.372241    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde" containerName="revoke-keys"
Oct 30 21:01:55.372507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.372255    4810 cpu_manager.go:414] "containerMap: removing container" podUID="b8c6b938-5c3f-4222-89cb-244426dc086e" containerName="revoke-keys"
Oct 30 21:01:55.372507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.372263    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c6b938-5c3f-4222-89cb-244426dc086e" containerName="revoke-keys"
Oct 30 21:01:55.372507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.372386    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde" containerName="revoke-keys"
Oct 30 21:01:55.372507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.372397    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c6b938-5c3f-4222-89cb-244426dc086e" containerName="revoke-keys"
Oct 30 21:01:55.382880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.382059    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"]
Oct 30 21:01:55.382880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.382172    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:55.404040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.398658    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:55.404040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.398761    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:01:55.633938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.610139    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv96j\" (UniqueName: \"kubernetes.io/projected/c95d7998-c053-4219-bb4d-b8e27b03dc9e-kube-api-access-vv96j\") pod \"maas-api-revoke-sub-04ce478b-lx5mw\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:55.633938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.610232    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04ce478b-lx5mw\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:55.633938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.610301    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04ce478b-lx5mw\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:55.731920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.731886    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv96j\" (UniqueName: \"kubernetes.io/projected/c95d7998-c053-4219-bb4d-b8e27b03dc9e-kube-api-access-vv96j\") pod \"maas-api-revoke-sub-04ce478b-lx5mw\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:55.732223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.732204    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04ce478b-lx5mw\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:55.732393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.732377    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04ce478b-lx5mw\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:55.737118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.733323    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04ce478b-lx5mw\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:55.741594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.741572    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04ce478b-lx5mw\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:55.792882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.782047    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv96j\" (UniqueName: \"kubernetes.io/projected/c95d7998-c053-4219-bb4d-b8e27b03dc9e-kube-api-access-vv96j\") pod \"maas-api-revoke-sub-04ce478b-lx5mw\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:55.816878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:55.800067    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:01:57.261888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:57.235377    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"]
Oct 30 21:01:57.375255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:57.287662    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:57.377005 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:01:57.349606    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95d7998_c053_4219_bb4d_b8e27b03dc9e.slice/crio-873d612a8f82e1e66967d343fca4472e0bd1cb79784284e53412d7e830b8b9ad WatchSource:0}: Error finding container 873d612a8f82e1e66967d343fca4472e0bd1cb79784284e53412d7e830b8b9ad: Status 404 returned error can't find the container with id 873d612a8f82e1e66967d343fca4472e0bd1cb79784284e53412d7e830b8b9ad
Oct 30 21:01:57.377005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:57.355028    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:57.451672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:57.445056    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw" event={"ID":"c95d7998-c053-4219-bb4d-b8e27b03dc9e","Type":"ContainerStarted","Data":"873d612a8f82e1e66967d343fca4472e0bd1cb79784284e53412d7e830b8b9ad"}
Oct 30 21:01:57.665884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:57.655629    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:58.470817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:58.469247    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw" event={"ID":"c95d7998-c053-4219-bb4d-b8e27b03dc9e","Type":"ContainerStarted","Data":"d8a03a9254d846801ed105a5561f2af3d93ba6c6f82b7337234d109d74177060"}
Oct 30 21:01:58.510920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:58.504956    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw" podStartSLOduration=3.504933475 podStartE2EDuration="3.504933475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:58.489950661 +0000 UTC m=+1484.981767217" watchObservedRunningTime="2026-10-30 21:01:58.504933475 +0000 UTC m=+1484.996750021"
Oct 30 21:01:59.479405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:59.479368    4810 generic.go:356] "Generic (PLEG): container finished" podID="c95d7998-c053-4219-bb4d-b8e27b03dc9e" containerID="d8a03a9254d846801ed105a5561f2af3d93ba6c6f82b7337234d109d74177060" exitCode=0
Oct 30 21:01:59.479932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:59.479460    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw" event={"ID":"c95d7998-c053-4219-bb4d-b8e27b03dc9e","Type":"ContainerDied","Data":"d8a03a9254d846801ed105a5561f2af3d93ba6c6f82b7337234d109d74177060"}
Oct 30 21:01:59.958705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:59.958670    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"]
Oct 30 21:01:59.962786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:59.962769    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:01:59.979821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:01:59.979790    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"]
Oct 30 21:02:00.033186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.033147    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctm2l\" (UniqueName: \"kubernetes.io/projected/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-kube-api-access-ctm2l\") pod \"maas-api-revoke-sub-1fdef763-l7ff7\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:00.033365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.033283    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fdef763-l7ff7\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:00.033365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.033337    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fdef763-l7ff7\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:00.134174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.134137    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctm2l\" (UniqueName: \"kubernetes.io/projected/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-kube-api-access-ctm2l\") pod \"maas-api-revoke-sub-1fdef763-l7ff7\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:00.134351 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.134203    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fdef763-l7ff7\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:00.134351 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.134234    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fdef763-l7ff7\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:00.135001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.134975    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fdef763-l7ff7\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:00.136637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.136613    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fdef763-l7ff7\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:00.145678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.145655    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctm2l\" (UniqueName: \"kubernetes.io/projected/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-kube-api-access-ctm2l\") pod \"maas-api-revoke-sub-1fdef763-l7ff7\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:00.280537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.280455    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:00.579396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.579340    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"]
Oct 30 21:02:00.836970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.826251    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:02:00.857810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.857770    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-openshift-ca\") pod \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") "
Oct 30 21:02:00.858039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.857950    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95d7998-c053-4219-bb4d-b8e27b03dc9e-kube-api-access-vv96j\" (UniqueName: \"kubernetes.io/projected/c95d7998-c053-4219-bb4d-b8e27b03dc9e-kube-api-access-vv96j\") pod \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") "
Oct 30 21:02:00.858039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.857997    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-xks-ca\") pod \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\" (UID: \"c95d7998-c053-4219-bb4d-b8e27b03dc9e\") "
Oct 30 21:02:00.859195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.859164    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-openshift-ca" pod "c95d7998-c053-4219-bb4d-b8e27b03dc9e" (UID: "c95d7998-c053-4219-bb4d-b8e27b03dc9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:00.862234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.861844    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95d7998-c053-4219-bb4d-b8e27b03dc9e-kube-api-access-vv96j" pod "c95d7998-c053-4219-bb4d-b8e27b03dc9e" (UID: "c95d7998-c053-4219-bb4d-b8e27b03dc9e"). InnerVolumeSpecName "kube-api-access-vv96j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:00.863462 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.863433    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-xks-ca" pod "c95d7998-c053-4219-bb4d-b8e27b03dc9e" (UID: "c95d7998-c053-4219-bb4d-b8e27b03dc9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:00.959387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.959351    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:00.959387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.959383    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv96j\" (UniqueName: \"kubernetes.io/projected/c95d7998-c053-4219-bb4d-b8e27b03dc9e-kube-api-access-vv96j\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:00.959387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:00.959393    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c95d7998-c053-4219-bb4d-b8e27b03dc9e-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:01.497975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:01.497889    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw" event={"ID":"c95d7998-c053-4219-bb4d-b8e27b03dc9e","Type":"ContainerDied","Data":"873d612a8f82e1e66967d343fca4472e0bd1cb79784284e53412d7e830b8b9ad"}
Oct 30 21:02:01.497975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:01.497927    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"
Oct 30 21:02:01.497975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:01.497954    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="873d612a8f82e1e66967d343fca4472e0bd1cb79784284e53412d7e830b8b9ad"
Oct 30 21:02:01.499791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:01.499767    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7" event={"ID":"ceec1e25-2da9-4a0d-9737-77e6b01b19a3","Type":"ContainerStarted","Data":"4843d126995e2dc45db141a686f6b078ec9711befd0b6dfa67fa3d63412a83da"}
Oct 30 21:02:01.499954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:01.499808    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7" event={"ID":"ceec1e25-2da9-4a0d-9737-77e6b01b19a3","Type":"ContainerStarted","Data":"ecad99d3aac44dce5aa4ea38e282c88c0edc5e397d3b53d31619a1ef9e7b6a34"}
Oct 30 21:02:01.537878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:01.523748    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7" podStartSLOduration=2.5237223909999997 podStartE2EDuration="2.523722391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:01.520240281 +0000 UTC m=+1488.012056828" watchObservedRunningTime="2026-10-30 21:02:01.523722391 +0000 UTC m=+1488.015538937"
Oct 30 21:02:02.508874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:02.508832    4810 generic.go:356] "Generic (PLEG): container finished" podID="ceec1e25-2da9-4a0d-9737-77e6b01b19a3" containerID="4843d126995e2dc45db141a686f6b078ec9711befd0b6dfa67fa3d63412a83da" exitCode=0
Oct 30 21:02:02.509270 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:02.508933    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7" event={"ID":"ceec1e25-2da9-4a0d-9737-77e6b01b19a3","Type":"ContainerDied","Data":"4843d126995e2dc45db141a686f6b078ec9711befd0b6dfa67fa3d63412a83da"}
Oct 30 21:02:03.688756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:03.688724    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:03.803397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:03.803313    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-kube-api-access-ctm2l\" (UniqueName: \"kubernetes.io/projected/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-kube-api-access-ctm2l\") pod \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") "
Oct 30 21:02:03.803397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:03.803364    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-openshift-ca\") pod \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") "
Oct 30 21:02:03.803647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:03.803404    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-xks-ca\") pod \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\" (UID: \"ceec1e25-2da9-4a0d-9737-77e6b01b19a3\") "
Oct 30 21:02:03.803837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:03.803813    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-openshift-ca" pod "ceec1e25-2da9-4a0d-9737-77e6b01b19a3" (UID: "ceec1e25-2da9-4a0d-9737-77e6b01b19a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:03.805667 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:03.805638    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-xks-ca" pod "ceec1e25-2da9-4a0d-9737-77e6b01b19a3" (UID: "ceec1e25-2da9-4a0d-9737-77e6b01b19a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:03.805778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:03.805763    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-kube-api-access-ctm2l" pod "ceec1e25-2da9-4a0d-9737-77e6b01b19a3" (UID: "ceec1e25-2da9-4a0d-9737-77e6b01b19a3"). InnerVolumeSpecName "kube-api-access-ctm2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:03.904580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:03.904540    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctm2l\" (UniqueName: \"kubernetes.io/projected/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-kube-api-access-ctm2l\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:03.904580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:03.904576    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:03.904783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:03.904590    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceec1e25-2da9-4a0d-9737-77e6b01b19a3-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:04.526558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:04.526527    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"
Oct 30 21:02:04.526558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:04.526543    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7" event={"ID":"ceec1e25-2da9-4a0d-9737-77e6b01b19a3","Type":"ContainerDied","Data":"ecad99d3aac44dce5aa4ea38e282c88c0edc5e397d3b53d31619a1ef9e7b6a34"}
Oct 30 21:02:04.526773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:04.526592    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecad99d3aac44dce5aa4ea38e282c88c0edc5e397d3b53d31619a1ef9e7b6a34"
Oct 30 21:02:10.834752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.834711    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"]
Oct 30 21:02:10.835335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.835319    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ceec1e25-2da9-4a0d-9737-77e6b01b19a3" containerName="revoke-keys"
Oct 30 21:02:10.835383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.835336    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceec1e25-2da9-4a0d-9737-77e6b01b19a3" containerName="revoke-keys"
Oct 30 21:02:10.835383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.835372    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c95d7998-c053-4219-bb4d-b8e27b03dc9e" containerName="revoke-keys"
Oct 30 21:02:10.835383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.835378    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95d7998-c053-4219-bb4d-b8e27b03dc9e" containerName="revoke-keys"
Oct 30 21:02:10.835478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.835466    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95d7998-c053-4219-bb4d-b8e27b03dc9e" containerName="revoke-keys"
Oct 30 21:02:10.835511 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.835479    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceec1e25-2da9-4a0d-9737-77e6b01b19a3" containerName="revoke-keys"
Oct 30 21:02:10.840387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.840368    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:10.845994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.845967    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:10.846572 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.846549    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:02:10.849032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.849008    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"]
Oct 30 21:02:10.920069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.920037    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-347fecc2-48prq\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:10.920243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.920080    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-347fecc2-48prq\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:10.920337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:10.920318    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7c6b\" (UniqueName: \"kubernetes.io/projected/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-kube-api-access-l7c6b\") pod \"maas-api-revoke-sub-347fecc2-48prq\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:11.021362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:11.021330    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-347fecc2-48prq\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:11.021552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:11.021374    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-347fecc2-48prq\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:11.021552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:11.021524    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7c6b\" (UniqueName: \"kubernetes.io/projected/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-kube-api-access-l7c6b\") pod \"maas-api-revoke-sub-347fecc2-48prq\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:11.021985 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:11.021964    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-347fecc2-48prq\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:11.023699 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:11.023680    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-347fecc2-48prq\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:11.030754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:11.030735    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7c6b\" (UniqueName: \"kubernetes.io/projected/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-kube-api-access-l7c6b\") pod \"maas-api-revoke-sub-347fecc2-48prq\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:11.158187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:11.158148    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:11.694676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:11.694157    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"]
Oct 30 21:02:12.628583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:12.628540    4810 generic.go:356] "Generic (PLEG): container finished" podID="080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2" containerID="090530abd905ef902823925b944d3a4e85aea003a0bd3821bf254ba1233ea945" exitCode=0
Oct 30 21:02:12.629056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:12.628965    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq" event={"ID":"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2","Type":"ContainerDied","Data":"090530abd905ef902823925b944d3a4e85aea003a0bd3821bf254ba1233ea945"}
Oct 30 21:02:12.629056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:12.629030    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq" event={"ID":"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2","Type":"ContainerStarted","Data":"cb34047d78cf811d7edbc7342992cc9539d9417f2082ca0dda998fd5740680f8"}
Oct 30 21:02:13.797056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:13.797029    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:13.816994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:13.816954    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-openshift-ca\") pod \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") "
Oct 30 21:02:13.817221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:13.817206    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-kube-api-access-l7c6b\" (UniqueName: \"kubernetes.io/projected/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-kube-api-access-l7c6b\") pod \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") "
Oct 30 21:02:13.817291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:13.817265    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-xks-ca\") pod \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\" (UID: \"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2\") "
Oct 30 21:02:13.818452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:13.818422    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-openshift-ca" pod "080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2" (UID: "080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:13.820399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:13.820374    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-xks-ca" pod "080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2" (UID: "080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:13.820768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:13.820741    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-kube-api-access-l7c6b" pod "080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2" (UID: "080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2"). InnerVolumeSpecName "kube-api-access-l7c6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:13.919106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:13.919022    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7c6b\" (UniqueName: \"kubernetes.io/projected/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-kube-api-access-l7c6b\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:13.919329 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:13.919309    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:13.919439 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:13.919424    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:14.312656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:14.312572    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:02:14.344135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:14.344111    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:02:14.646614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:14.646589    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"
Oct 30 21:02:14.646783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:14.646633    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq" event={"ID":"080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2","Type":"ContainerDied","Data":"cb34047d78cf811d7edbc7342992cc9539d9417f2082ca0dda998fd5740680f8"}
Oct 30 21:02:14.646783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:14.646703    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb34047d78cf811d7edbc7342992cc9539d9417f2082ca0dda998fd5740680f8"
Oct 30 21:02:15.287387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:15.287351    4810 scope.go:122] "RemoveContainer" containerID="051cffc0fc99e0b4c912cb8cf3b9fddf73bca8e0719bf1432439b29c82d3acce"
Oct 30 21:02:15.299258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:15.299148    4810 scope.go:122] "RemoveContainer" containerID="57b6fc5b9b44b523073def07df197be1adfdd57df55b9c3cb0bb38d3a9c86df1"
Oct 30 21:02:15.766233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:15.766200    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:02:16.824470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.824430    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"]
Oct 30 21:02:16.825378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.825354    4810 cpu_manager.go:414] "containerMap: removing container" podUID="080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2" containerName="revoke-keys"
Oct 30 21:02:16.825378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.825377    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2" containerName="revoke-keys"
Oct 30 21:02:16.825570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.825558    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2" containerName="revoke-keys"
Oct 30 21:02:16.830977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.830951    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:02:16.840442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.840420    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-8fa15b-dockercfg-w2c8n\""
Oct 30 21:02:16.846683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.846658    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"]
Oct 30 21:02:16.852691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.852670    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-8fa15b\""
Oct 30 21:02:16.870737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.870706    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9846a273-e5af-4bfc-b365-0d3fb804e5c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd\" (UID: \"9846a273-e5af-4bfc-b365-0d3fb804e5c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:02:16.870875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.870826    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgkcd\" (UniqueName: \"kubernetes.io/projected/9846a273-e5af-4bfc-b365-0d3fb804e5c4-kube-api-access-hgkcd\") pod \"payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd\" (UID: \"9846a273-e5af-4bfc-b365-0d3fb804e5c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:02:16.971741 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.971700    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9846a273-e5af-4bfc-b365-0d3fb804e5c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd\" (UID: \"9846a273-e5af-4bfc-b365-0d3fb804e5c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:02:16.971977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.971769    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgkcd\" (UniqueName: \"kubernetes.io/projected/9846a273-e5af-4bfc-b365-0d3fb804e5c4-kube-api-access-hgkcd\") pod \"payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd\" (UID: \"9846a273-e5af-4bfc-b365-0d3fb804e5c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:02:16.972525 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.972498    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9846a273-e5af-4bfc-b365-0d3fb804e5c4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd\" (UID: \"9846a273-e5af-4bfc-b365-0d3fb804e5c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:02:16.984010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:16.983983    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgkcd\" (UniqueName: \"kubernetes.io/projected/9846a273-e5af-4bfc-b365-0d3fb804e5c4-kube-api-access-hgkcd\") pod \"payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd\" (UID: \"9846a273-e5af-4bfc-b365-0d3fb804e5c4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:02:17.149616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:17.149288    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:02:18.189812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:18.189767    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"]
Oct 30 21:02:18.955632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:18.955575    4810 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:19.102276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:19.100921    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd" event={"ID":"9846a273-e5af-4bfc-b365-0d3fb804e5c4","Type":"ContainerStarted","Data":"8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe"}
Oct 30 21:02:19.102276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:19.100993    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd" event={"ID":"9846a273-e5af-4bfc-b365-0d3fb804e5c4","Type":"ContainerStarted","Data":"adb712f54a1c17487b4e6410daa934019fe9daa9e0b5a81bb6192f5ac611d935"}
Oct 30 21:02:19.112223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:19.110089    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:02:19.186880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:19.186296    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd" podStartSLOduration=3.186274046 podStartE2EDuration="3.186274046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:19.16731121 +0000 UTC m=+1505.659127759" watchObservedRunningTime="2026-10-30 21:02:19.186274046 +0000 UTC m=+1505.678090592"
Oct 30 21:02:21.993298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:21.993264    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"]
Oct 30 21:02:22.003182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.002562    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.009879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.006621    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:22.009879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.007644    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:02:22.009879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.008976    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"]
Oct 30 21:02:22.120393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.120357    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9298d236-251a-4840-b72f-3438a05960df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.121018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.120976    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvhl\" (UniqueName: \"kubernetes.io/projected/9298d236-251a-4840-b72f-3438a05960df-kube-api-access-5lvhl\") pod \"maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.121298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.121272    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9298d236-251a-4840-b72f-3438a05960df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.222651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.222612    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9298d236-251a-4840-b72f-3438a05960df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.222851 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.222818    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9298d236-251a-4840-b72f-3438a05960df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.222851 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.222852    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvhl\" (UniqueName: \"kubernetes.io/projected/9298d236-251a-4840-b72f-3438a05960df-kube-api-access-5lvhl\") pod \"maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.223406 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.223383    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9298d236-251a-4840-b72f-3438a05960df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.225310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.225282    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9298d236-251a-4840-b72f-3438a05960df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.234266 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.234244    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvhl\" (UniqueName: \"kubernetes.io/projected/9298d236-251a-4840-b72f-3438a05960df-kube-api-access-5lvhl\") pod \"maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.332096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.332014    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:22.738931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:22.738902    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"]
Oct 30 21:02:22.740094 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:02:22.740062    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9298d236_251a_4840_b72f_3438a05960df.slice/crio-319cfdeb0f5ff448436bf76883725525d255bc61f3ed6cf5fb670e3d9fe29a34 WatchSource:0}: Error finding container 319cfdeb0f5ff448436bf76883725525d255bc61f3ed6cf5fb670e3d9fe29a34: Status 404 returned error can't find the container with id 319cfdeb0f5ff448436bf76883725525d255bc61f3ed6cf5fb670e3d9fe29a34
Oct 30 21:02:23.209839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:23.208716    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf" event={"ID":"9298d236-251a-4840-b72f-3438a05960df","Type":"ContainerStarted","Data":"601b88ad1b30e885f2c5fc686f6ecdb6590553a4635695a68378ce4e3280aa63"}
Oct 30 21:02:23.209839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:23.208787    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf" event={"ID":"9298d236-251a-4840-b72f-3438a05960df","Type":"ContainerStarted","Data":"319cfdeb0f5ff448436bf76883725525d255bc61f3ed6cf5fb670e3d9fe29a34"}
Oct 30 21:02:23.250367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:23.250282    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf" podStartSLOduration=2.250259938 podStartE2EDuration="2.250259938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:23.231243827 +0000 UTC m=+1509.723060373" watchObservedRunningTime="2026-10-30 21:02:23.250259938 +0000 UTC m=+1509.742076484"
Oct 30 21:02:24.219500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:24.219464    4810 generic.go:356] "Generic (PLEG): container finished" podID="9298d236-251a-4840-b72f-3438a05960df" containerID="601b88ad1b30e885f2c5fc686f6ecdb6590553a4635695a68378ce4e3280aa63" exitCode=0
Oct 30 21:02:24.219930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:24.219553    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf" event={"ID":"9298d236-251a-4840-b72f-3438a05960df","Type":"ContainerDied","Data":"601b88ad1b30e885f2c5fc686f6ecdb6590553a4635695a68378ce4e3280aa63"}
Oct 30 21:02:25.313246 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:02:25.313209    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:02:25.520092 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:25.520066    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:25.623812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:25.623735    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9298d236-251a-4840-b72f-3438a05960df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9298d236-251a-4840-b72f-3438a05960df-maas-api-xks-ca\") pod \"9298d236-251a-4840-b72f-3438a05960df\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") "
Oct 30 21:02:25.623812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:25.623798    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9298d236-251a-4840-b72f-3438a05960df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9298d236-251a-4840-b72f-3438a05960df-maas-api-openshift-ca\") pod \"9298d236-251a-4840-b72f-3438a05960df\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") "
Oct 30 21:02:25.624088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:25.623996    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9298d236-251a-4840-b72f-3438a05960df-kube-api-access-5lvhl\" (UniqueName: \"kubernetes.io/projected/9298d236-251a-4840-b72f-3438a05960df-kube-api-access-5lvhl\") pod \"9298d236-251a-4840-b72f-3438a05960df\" (UID: \"9298d236-251a-4840-b72f-3438a05960df\") "
Oct 30 21:02:25.624223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:25.624200    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9298d236-251a-4840-b72f-3438a05960df-maas-api-openshift-ca" pod "9298d236-251a-4840-b72f-3438a05960df" (UID: "9298d236-251a-4840-b72f-3438a05960df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:25.624417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:25.624400    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9298d236-251a-4840-b72f-3438a05960df-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:25.625837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:25.625805    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9298d236-251a-4840-b72f-3438a05960df-maas-api-xks-ca" pod "9298d236-251a-4840-b72f-3438a05960df" (UID: "9298d236-251a-4840-b72f-3438a05960df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:25.626034 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:25.626014    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9298d236-251a-4840-b72f-3438a05960df-kube-api-access-5lvhl" pod "9298d236-251a-4840-b72f-3438a05960df" (UID: "9298d236-251a-4840-b72f-3438a05960df"). InnerVolumeSpecName "kube-api-access-5lvhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:25.725560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:25.725523    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9298d236-251a-4840-b72f-3438a05960df-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:25.725560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:25.725552    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvhl\" (UniqueName: \"kubernetes.io/projected/9298d236-251a-4840-b72f-3438a05960df-kube-api-access-5lvhl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:26.373115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:26.373069    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf" event={"ID":"9298d236-251a-4840-b72f-3438a05960df","Type":"ContainerDied","Data":"319cfdeb0f5ff448436bf76883725525d255bc61f3ed6cf5fb670e3d9fe29a34"}
Oct 30 21:02:26.373622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:26.373196    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="319cfdeb0f5ff448436bf76883725525d255bc61f3ed6cf5fb670e3d9fe29a34"
Oct 30 21:02:26.373622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:26.373294    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"
Oct 30 21:02:27.355238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.355206    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"]
Oct 30 21:02:27.355808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.355793    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9298d236-251a-4840-b72f-3438a05960df" containerName="revoke-keys"
Oct 30 21:02:27.355905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.355808    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9298d236-251a-4840-b72f-3438a05960df" containerName="revoke-keys"
Oct 30 21:02:27.355973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.355960    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9298d236-251a-4840-b72f-3438a05960df" containerName="revoke-keys"
Oct 30 21:02:27.359656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.359637    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:02:27.363911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.363887    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-e53c37\""
Oct 30 21:02:27.369218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.369037    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"]
Oct 30 21:02:27.392410 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.392386    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"]
Oct 30 21:02:27.396774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.396755    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:02:27.412003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.411979    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"]
Oct 30 21:02:27.463072 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.463042    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-e53c37-dockercfg-9sx4b\""
Oct 30 21:02:27.530958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.530915    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212bd647-b861-4804-88a0-a3cd4372ece2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4\" (UID: \"212bd647-b861-4804-88a0-a3cd4372ece2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:02:27.531316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.531284    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktcs\" (UniqueName: \"kubernetes.io/projected/212bd647-b861-4804-88a0-a3cd4372ece2-kube-api-access-xktcs\") pod \"payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4\" (UID: \"212bd647-b861-4804-88a0-a3cd4372ece2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:02:27.531550 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.531530    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e74b3bb-da56-4fe7-9585-916338363efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst\" (UID: \"0e74b3bb-da56-4fe7-9585-916338363efb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:02:27.531744 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.531707    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgkxf\" (UniqueName: \"kubernetes.io/projected/0e74b3bb-da56-4fe7-9585-916338363efb-kube-api-access-vgkxf\") pod \"payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst\" (UID: \"0e74b3bb-da56-4fe7-9585-916338363efb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:02:27.633170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.633126    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212bd647-b861-4804-88a0-a3cd4372ece2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4\" (UID: \"212bd647-b861-4804-88a0-a3cd4372ece2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:02:27.633374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.633182    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktcs\" (UniqueName: \"kubernetes.io/projected/212bd647-b861-4804-88a0-a3cd4372ece2-kube-api-access-xktcs\") pod \"payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4\" (UID: \"212bd647-b861-4804-88a0-a3cd4372ece2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:02:27.633374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.633237    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e74b3bb-da56-4fe7-9585-916338363efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst\" (UID: \"0e74b3bb-da56-4fe7-9585-916338363efb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:02:27.633374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.633287    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgkxf\" (UniqueName: \"kubernetes.io/projected/0e74b3bb-da56-4fe7-9585-916338363efb-kube-api-access-vgkxf\") pod \"payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst\" (UID: \"0e74b3bb-da56-4fe7-9585-916338363efb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:02:27.635918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.635870    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e74b3bb-da56-4fe7-9585-916338363efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst\" (UID: \"0e74b3bb-da56-4fe7-9585-916338363efb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:02:27.637607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.637580    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212bd647-b861-4804-88a0-a3cd4372ece2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4\" (UID: \"212bd647-b861-4804-88a0-a3cd4372ece2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:02:27.651225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.648991    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgkxf\" (UniqueName: \"kubernetes.io/projected/0e74b3bb-da56-4fe7-9585-916338363efb-kube-api-access-vgkxf\") pod \"payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst\" (UID: \"0e74b3bb-da56-4fe7-9585-916338363efb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:02:27.655778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.655752    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktcs\" (UniqueName: \"kubernetes.io/projected/212bd647-b861-4804-88a0-a3cd4372ece2-kube-api-access-xktcs\") pod \"payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4\" (UID: \"212bd647-b861-4804-88a0-a3cd4372ece2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:02:27.680913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.680884    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:02:27.717350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.717322    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:02:27.974312 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:02:27.974284    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212bd647_b861_4804_88a0_a3cd4372ece2.slice/crio-8098e9da60e9153b7c47721da3c01ebb3e6b52f98fb9dd60764a55068cb9ba56 WatchSource:0}: Error finding container 8098e9da60e9153b7c47721da3c01ebb3e6b52f98fb9dd60764a55068cb9ba56: Status 404 returned error can't find the container with id 8098e9da60e9153b7c47721da3c01ebb3e6b52f98fb9dd60764a55068cb9ba56
Oct 30 21:02:27.976097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:27.976068    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"]
Oct 30 21:02:28.077224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:28.077197    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"]
Oct 30 21:02:28.398010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:28.397974    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst" event={"ID":"0e74b3bb-da56-4fe7-9585-916338363efb","Type":"ContainerStarted","Data":"5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3"}
Oct 30 21:02:28.398488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:28.398040    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst" event={"ID":"0e74b3bb-da56-4fe7-9585-916338363efb","Type":"ContainerStarted","Data":"8d7e13011d8430884810eb572392d0949ac93f4a1f618364a990267a3888e22f"}
Oct 30 21:02:28.398488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:28.398071    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:02:28.399917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:28.399890    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4" event={"ID":"212bd647-b861-4804-88a0-a3cd4372ece2","Type":"ContainerStarted","Data":"de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8"}
Oct 30 21:02:28.400049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:28.399941    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4" event={"ID":"212bd647-b861-4804-88a0-a3cd4372ece2","Type":"ContainerStarted","Data":"8098e9da60e9153b7c47721da3c01ebb3e6b52f98fb9dd60764a55068cb9ba56"}
Oct 30 21:02:28.400049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:28.399981    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:02:28.423654 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:28.423609    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst" podStartSLOduration=1.423596101 podStartE2EDuration="1.423596101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:28.420279604 +0000 UTC m=+1514.912096153" watchObservedRunningTime="2026-10-30 21:02:28.423596101 +0000 UTC m=+1514.915412656"
Oct 30 21:02:28.445697 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:28.445653    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4" podStartSLOduration=1.44564016 podStartE2EDuration="1.44564016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:28.442217708 +0000 UTC m=+1514.934034254" watchObservedRunningTime="2026-10-30 21:02:28.44564016 +0000 UTC m=+1514.937456705"
Oct 30 21:02:29.517513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.517477    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"]
Oct 30 21:02:29.523133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.523108    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:02:29.531019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.530993    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"]
Oct 30 21:02:29.537381 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.532382    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-74f9f4\""
Oct 30 21:02:29.537381 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.532520    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-74f9f4-dockercfg-lsbmp\""
Oct 30 21:02:29.576321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.576277    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"]
Oct 30 21:02:29.580802 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.580713    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:02:29.592949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.592775    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"]
Oct 30 21:02:29.672631 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.672578    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ss8\" (UniqueName: \"kubernetes.io/projected/e4f5604e-38c2-4f39-9f51-496b1a383cc9-kube-api-access-57ss8\") pod \"payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l\" (UID: \"e4f5604e-38c2-4f39-9f51-496b1a383cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:02:29.674284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.674229    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f5604e-38c2-4f39-9f51-496b1a383cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l\" (UID: \"e4f5604e-38c2-4f39-9f51-496b1a383cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:02:29.775526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.775423    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f5604e-38c2-4f39-9f51-496b1a383cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l\" (UID: \"e4f5604e-38c2-4f39-9f51-496b1a383cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:02:29.775526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.775527    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfp47\" (UniqueName: \"kubernetes.io/projected/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-kube-api-access-sfp47\") pod \"payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm\" (UID: \"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:02:29.775785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.775567    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm\" (UID: \"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:02:29.775785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.775595    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57ss8\" (UniqueName: \"kubernetes.io/projected/e4f5604e-38c2-4f39-9f51-496b1a383cc9-kube-api-access-57ss8\") pod \"payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l\" (UID: \"e4f5604e-38c2-4f39-9f51-496b1a383cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:02:29.776579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.776555    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f5604e-38c2-4f39-9f51-496b1a383cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l\" (UID: \"e4f5604e-38c2-4f39-9f51-496b1a383cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:02:29.788557 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.788524    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ss8\" (UniqueName: \"kubernetes.io/projected/e4f5604e-38c2-4f39-9f51-496b1a383cc9-kube-api-access-57ss8\") pod \"payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l\" (UID: \"e4f5604e-38c2-4f39-9f51-496b1a383cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:02:29.853194 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.853153    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:02:29.878503 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.877929    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfp47\" (UniqueName: \"kubernetes.io/projected/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-kube-api-access-sfp47\") pod \"payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm\" (UID: \"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:02:29.878503 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.878027    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm\" (UID: \"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:02:29.879371 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.879331    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm\" (UID: \"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:02:29.895583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.894626    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfp47\" (UniqueName: \"kubernetes.io/projected/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-kube-api-access-sfp47\") pod \"payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm\" (UID: \"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:02:29.904200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:29.904170    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:02:30.163460 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.163435    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"]
Oct 30 21:02:30.164046 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:02:30.164019    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f5604e_38c2_4f39_9f51_496b1a383cc9.slice/crio-0e9dcae8e3249294e2474e803a9400da91f194bbc6f0dea75ed37d4b8a4f88bd WatchSource:0}: Error finding container 0e9dcae8e3249294e2474e803a9400da91f194bbc6f0dea75ed37d4b8a4f88bd: Status 404 returned error can't find the container with id 0e9dcae8e3249294e2474e803a9400da91f194bbc6f0dea75ed37d4b8a4f88bd
Oct 30 21:02:30.210833 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:02:30.210805    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e70ac49_535d_4ba4_af02_4f15b3e4ed5b.slice/crio-9b21cb3f097d31629af0c84bf48e86e53b51e67f112eabf47595607e3855628e WatchSource:0}: Error finding container 9b21cb3f097d31629af0c84bf48e86e53b51e67f112eabf47595607e3855628e: Status 404 returned error can't find the container with id 9b21cb3f097d31629af0c84bf48e86e53b51e67f112eabf47595607e3855628e
Oct 30 21:02:30.211340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.211316    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"]
Oct 30 21:02:30.349425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.349378    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"]
Oct 30 21:02:30.359186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.359158    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.359633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.359207    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"]
Oct 30 21:02:30.404895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.404850    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:30.405089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.405071    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:02:30.424928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.424828    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm" event={"ID":"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b","Type":"ContainerStarted","Data":"5c190ee7d3a948ae65fd41b6fbcb9fd0796ba304739be52a757dbe5c6b55b2c8"}
Oct 30 21:02:30.424928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.424901    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm" event={"ID":"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b","Type":"ContainerStarted","Data":"9b21cb3f097d31629af0c84bf48e86e53b51e67f112eabf47595607e3855628e"}
Oct 30 21:02:30.425145 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.424939    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:02:30.430501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.430469    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l" event={"ID":"e4f5604e-38c2-4f39-9f51-496b1a383cc9","Type":"ContainerStarted","Data":"b660eaaf20d9eff2dd70c643e0409caee011b9da0275a2a2dac2836b62178cc1"}
Oct 30 21:02:30.430670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.430523    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l" event={"ID":"e4f5604e-38c2-4f39-9f51-496b1a383cc9","Type":"ContainerStarted","Data":"0e9dcae8e3249294e2474e803a9400da91f194bbc6f0dea75ed37d4b8a4f88bd"}
Oct 30 21:02:30.430670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.430550    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:02:30.438164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.438135    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.438343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.438322    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.438528 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.438508    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvfv\" (UniqueName: \"kubernetes.io/projected/f9127c3f-272b-4144-aef6-234d64f12339-kube-api-access-kmvfv\") pod \"maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.446042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.446001    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm" podStartSLOduration=1.445989001 podStartE2EDuration="1.445989001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:30.443354155 +0000 UTC m=+1516.935170722" watchObservedRunningTime="2026-10-30 21:02:30.445989001 +0000 UTC m=+1516.937805559"
Oct 30 21:02:30.474608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.474568    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l" podStartSLOduration=1.474546894 podStartE2EDuration="1.474546894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:30.47114456 +0000 UTC m=+1516.962961105" watchObservedRunningTime="2026-10-30 21:02:30.474546894 +0000 UTC m=+1516.966363483"
Oct 30 21:02:30.544374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.544341    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.544837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.544543    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.544837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.544710    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvfv\" (UniqueName: \"kubernetes.io/projected/f9127c3f-272b-4144-aef6-234d64f12339-kube-api-access-kmvfv\") pod \"maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.546044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.546019    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.548396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.548369    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.553980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.553956    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvfv\" (UniqueName: \"kubernetes.io/projected/f9127c3f-272b-4144-aef6-234d64f12339-kube-api-access-kmvfv\") pod \"maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.681925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.681612    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:30.972531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:30.972500    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"]
Oct 30 21:02:30.973272 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:02:30.973240    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9127c3f_272b_4144_aef6_234d64f12339.slice/crio-1889b9eaa42d19f258caad8a0f1b1b13f49bb7d1366dacc512d2e3cd0e3db3c2 WatchSource:0}: Error finding container 1889b9eaa42d19f258caad8a0f1b1b13f49bb7d1366dacc512d2e3cd0e3db3c2: Status 404 returned error can't find the container with id 1889b9eaa42d19f258caad8a0f1b1b13f49bb7d1366dacc512d2e3cd0e3db3c2
Oct 30 21:02:31.140314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:31.140290    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:02:31.443923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:31.443887    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj" event={"ID":"f9127c3f-272b-4144-aef6-234d64f12339","Type":"ContainerStarted","Data":"760c2e0de17fdb1503f62a5a1f70d20fc514b3e5d60b0a690f563416fcbac35b"}
Oct 30 21:02:31.444127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:31.443946    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj" event={"ID":"f9127c3f-272b-4144-aef6-234d64f12339","Type":"ContainerStarted","Data":"1889b9eaa42d19f258caad8a0f1b1b13f49bb7d1366dacc512d2e3cd0e3db3c2"}
Oct 30 21:02:31.464297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:31.464247    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj" podStartSLOduration=1.464233415 podStartE2EDuration="1.464233415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:31.460484276 +0000 UTC m=+1517.952300822" watchObservedRunningTime="2026-10-30 21:02:31.464233415 +0000 UTC m=+1517.956049960"
Oct 30 21:02:32.453820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:32.453785    4810 generic.go:356] "Generic (PLEG): container finished" podID="f9127c3f-272b-4144-aef6-234d64f12339" containerID="760c2e0de17fdb1503f62a5a1f70d20fc514b3e5d60b0a690f563416fcbac35b" exitCode=0
Oct 30 21:02:32.454235 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:32.453893    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj" event={"ID":"f9127c3f-272b-4144-aef6-234d64f12339","Type":"ContainerDied","Data":"760c2e0de17fdb1503f62a5a1f70d20fc514b3e5d60b0a690f563416fcbac35b"}
Oct 30 21:02:33.641899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:33.641840    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:33.801913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:33.801791    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9127c3f-272b-4144-aef6-234d64f12339-kube-api-access-kmvfv\" (UniqueName: \"kubernetes.io/projected/f9127c3f-272b-4144-aef6-234d64f12339-kube-api-access-kmvfv\") pod \"f9127c3f-272b-4144-aef6-234d64f12339\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") "
Oct 30 21:02:33.801913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:33.801837    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-openshift-ca\") pod \"f9127c3f-272b-4144-aef6-234d64f12339\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") "
Oct 30 21:02:33.801913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:33.801901    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-xks-ca\") pod \"f9127c3f-272b-4144-aef6-234d64f12339\" (UID: \"f9127c3f-272b-4144-aef6-234d64f12339\") "
Oct 30 21:02:33.802206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:33.802183    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-openshift-ca" pod "f9127c3f-272b-4144-aef6-234d64f12339" (UID: "f9127c3f-272b-4144-aef6-234d64f12339"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:33.802555 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:33.802525    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:33.804202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:33.804172    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9127c3f-272b-4144-aef6-234d64f12339-kube-api-access-kmvfv" pod "f9127c3f-272b-4144-aef6-234d64f12339" (UID: "f9127c3f-272b-4144-aef6-234d64f12339"). InnerVolumeSpecName "kube-api-access-kmvfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:33.804292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:33.804214    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-xks-ca" pod "f9127c3f-272b-4144-aef6-234d64f12339" (UID: "f9127c3f-272b-4144-aef6-234d64f12339"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:33.904157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:33.904123    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvfv\" (UniqueName: \"kubernetes.io/projected/f9127c3f-272b-4144-aef6-234d64f12339-kube-api-access-kmvfv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:33.904157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:33.904153    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9127c3f-272b-4144-aef6-234d64f12339-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.470309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:34.470279    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"
Oct 30 21:02:34.470483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:34.470285    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj" event={"ID":"f9127c3f-272b-4144-aef6-234d64f12339","Type":"ContainerDied","Data":"1889b9eaa42d19f258caad8a0f1b1b13f49bb7d1366dacc512d2e3cd0e3db3c2"}
Oct 30 21:02:34.470483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:34.470401    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1889b9eaa42d19f258caad8a0f1b1b13f49bb7d1366dacc512d2e3cd0e3db3c2"
Oct 30 21:02:35.640354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:35.640321    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"]
Oct 30 21:02:35.648137 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:35.648092    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-117ee1-24spf"]
Oct 30 21:02:35.667568 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:35.667539    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9298d236-251a-4840-b72f-3438a05960df" path="/var/lib/kubelet/pods/9298d236-251a-4840-b72f-3438a05960df/volumes"
Oct 30 21:02:39.417932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:39.417898    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:02:39.420595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:39.420541    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:02:41.446949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:41.446916    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:02:41.447428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:41.447109    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:02:42.376922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:42.376883    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:02:44.912941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.912904    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"]
Oct 30 21:02:44.914460 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.914429    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f9127c3f-272b-4144-aef6-234d64f12339" containerName="revoke-keys"
Oct 30 21:02:44.914649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.914635    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9127c3f-272b-4144-aef6-234d64f12339" containerName="revoke-keys"
Oct 30 21:02:44.915820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.915799    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9127c3f-272b-4144-aef6-234d64f12339" containerName="revoke-keys"
Oct 30 21:02:44.940071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.940046    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"]
Oct 30 21:02:44.940320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.940306    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:44.955249 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.955223    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:02:44.955446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.955433    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:44.994418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.994309    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:44.994973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.994947    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7gdz\" (UniqueName: \"kubernetes.io/projected/803b054b-e6d5-47ed-b254-75a8cf95d38a-kube-api-access-h7gdz\") pod \"maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:44.998878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:44.995207    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:45.159671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:45.159543    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:45.161193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:45.160998    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7gdz\" (UniqueName: \"kubernetes.io/projected/803b054b-e6d5-47ed-b254-75a8cf95d38a-kube-api-access-h7gdz\") pod \"maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:45.182962 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:45.161806    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:45.182962 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:45.162682    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:45.223507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:45.223436    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:45.308346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:45.290998    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7gdz\" (UniqueName: \"kubernetes.io/projected/803b054b-e6d5-47ed-b254-75a8cf95d38a-kube-api-access-h7gdz\") pod \"maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:45.570488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:45.569276    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:45.613883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:45.571588    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:46.258527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:46.258501    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"]
Oct 30 21:02:46.259633 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:02:46.259608    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803b054b_e6d5_47ed_b254_75a8cf95d38a.slice/crio-036de691836405087a13cc72b86105d96e61ddb2f61581345795c86cfe9acafc WatchSource:0}: Error finding container 036de691836405087a13cc72b86105d96e61ddb2f61581345795c86cfe9acafc: Status 404 returned error can't find the container with id 036de691836405087a13cc72b86105d96e61ddb2f61581345795c86cfe9acafc
Oct 30 21:02:46.921543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:46.921493    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj" event={"ID":"803b054b-e6d5-47ed-b254-75a8cf95d38a","Type":"ContainerStarted","Data":"d0bd8e5e14063d5b952ebfddf9f64a5c30b35999e3156d0d0bc6b0bdf843bfc8"}
Oct 30 21:02:46.921715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:46.921563    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj" event={"ID":"803b054b-e6d5-47ed-b254-75a8cf95d38a","Type":"ContainerStarted","Data":"036de691836405087a13cc72b86105d96e61ddb2f61581345795c86cfe9acafc"}
Oct 30 21:02:46.941112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:46.941019    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj" podStartSLOduration=2.941005954 podStartE2EDuration="2.941005954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:46.937617599 +0000 UTC m=+1533.429434144" watchObservedRunningTime="2026-10-30 21:02:46.941005954 +0000 UTC m=+1533.432822554"
Oct 30 21:02:47.938503 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:47.938459    4810 generic.go:356] "Generic (PLEG): container finished" podID="803b054b-e6d5-47ed-b254-75a8cf95d38a" containerID="d0bd8e5e14063d5b952ebfddf9f64a5c30b35999e3156d0d0bc6b0bdf843bfc8" exitCode=0
Oct 30 21:02:47.939017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:47.938508    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj" event={"ID":"803b054b-e6d5-47ed-b254-75a8cf95d38a","Type":"ContainerDied","Data":"d0bd8e5e14063d5b952ebfddf9f64a5c30b35999e3156d0d0bc6b0bdf843bfc8"}
Oct 30 21:02:49.128204 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.128178    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:49.168217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.168189    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-openshift-ca\") pod \"803b054b-e6d5-47ed-b254-75a8cf95d38a\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") "
Oct 30 21:02:49.168401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.168388    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803b054b-e6d5-47ed-b254-75a8cf95d38a-kube-api-access-h7gdz\" (UniqueName: \"kubernetes.io/projected/803b054b-e6d5-47ed-b254-75a8cf95d38a-kube-api-access-h7gdz\") pod \"803b054b-e6d5-47ed-b254-75a8cf95d38a\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") "
Oct 30 21:02:49.168514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.168500    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-xks-ca\") pod \"803b054b-e6d5-47ed-b254-75a8cf95d38a\" (UID: \"803b054b-e6d5-47ed-b254-75a8cf95d38a\") "
Oct 30 21:02:49.168643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.168616    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-openshift-ca" pod "803b054b-e6d5-47ed-b254-75a8cf95d38a" (UID: "803b054b-e6d5-47ed-b254-75a8cf95d38a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:49.169026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.169002    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:49.171026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.171001    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-xks-ca" pod "803b054b-e6d5-47ed-b254-75a8cf95d38a" (UID: "803b054b-e6d5-47ed-b254-75a8cf95d38a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:49.171135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.171042    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803b054b-e6d5-47ed-b254-75a8cf95d38a-kube-api-access-h7gdz" pod "803b054b-e6d5-47ed-b254-75a8cf95d38a" (UID: "803b054b-e6d5-47ed-b254-75a8cf95d38a"). InnerVolumeSpecName "kube-api-access-h7gdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:49.270448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.270361    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803b054b-e6d5-47ed-b254-75a8cf95d38a-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:49.270448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.270389    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7gdz\" (UniqueName: \"kubernetes.io/projected/803b054b-e6d5-47ed-b254-75a8cf95d38a-kube-api-access-h7gdz\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:49.957004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.956918    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj" event={"ID":"803b054b-e6d5-47ed-b254-75a8cf95d38a","Type":"ContainerDied","Data":"036de691836405087a13cc72b86105d96e61ddb2f61581345795c86cfe9acafc"}
Oct 30 21:02:49.957395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.957374    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="036de691836405087a13cc72b86105d96e61ddb2f61581345795c86cfe9acafc"
Oct 30 21:02:49.957563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:49.957320    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"
Oct 30 21:02:50.093347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.093314    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"]
Oct 30 21:02:50.094043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.094025    4810 cpu_manager.go:414] "containerMap: removing container" podUID="803b054b-e6d5-47ed-b254-75a8cf95d38a" containerName="revoke-keys"
Oct 30 21:02:50.094121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.094044    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803b054b-e6d5-47ed-b254-75a8cf95d38a" containerName="revoke-keys"
Oct 30 21:02:50.094170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.094154    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="803b054b-e6d5-47ed-b254-75a8cf95d38a" containerName="revoke-keys"
Oct 30 21:02:50.097795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.097775    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.102331 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.102284    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:50.102331 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.102316    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:02:50.109732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.109680    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"]
Oct 30 21:02:50.183370 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.183338    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87efb90c-hsv4r\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.183891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.183389    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87efb90c-hsv4r\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.183891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.183618    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7vxv\" (UniqueName: \"kubernetes.io/projected/e6d4776d-e845-4fd5-bfab-0fd94598c75d-kube-api-access-b7vxv\") pod \"maas-api-revoke-sub-87efb90c-hsv4r\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.286943 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.286831    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vxv\" (UniqueName: \"kubernetes.io/projected/e6d4776d-e845-4fd5-bfab-0fd94598c75d-kube-api-access-b7vxv\") pod \"maas-api-revoke-sub-87efb90c-hsv4r\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.287122 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.287028    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87efb90c-hsv4r\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.287122 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.287083    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87efb90c-hsv4r\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.287936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.287908    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87efb90c-hsv4r\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.289916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.289868    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87efb90c-hsv4r\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.298205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.298181    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7vxv\" (UniqueName: \"kubernetes.io/projected/e6d4776d-e845-4fd5-bfab-0fd94598c75d-kube-api-access-b7vxv\") pod \"maas-api-revoke-sub-87efb90c-hsv4r\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.430393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.430338    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:50.735017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.734986    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"]
Oct 30 21:02:50.735165 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:02:50.734985    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d4776d_e845_4fd5_bfab_0fd94598c75d.slice/crio-86238d99b7a2515050b85ac6bb30501d90277bf1b36630c7a296e21307a7e7f6 WatchSource:0}: Error finding container 86238d99b7a2515050b85ac6bb30501d90277bf1b36630c7a296e21307a7e7f6: Status 404 returned error can't find the container with id 86238d99b7a2515050b85ac6bb30501d90277bf1b36630c7a296e21307a7e7f6
Oct 30 21:02:50.966081 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:50.966041    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r" event={"ID":"e6d4776d-e845-4fd5-bfab-0fd94598c75d","Type":"ContainerStarted","Data":"86238d99b7a2515050b85ac6bb30501d90277bf1b36630c7a296e21307a7e7f6"}
Oct 30 21:02:51.989744 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:51.989693    4810 generic.go:356] "Generic (PLEG): container finished" podID="e6d4776d-e845-4fd5-bfab-0fd94598c75d" containerID="0137e7384c8720feb37f87a9eb03921df3d0bbdc081886a04cc3516ab749d296" exitCode=0
Oct 30 21:02:51.990669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:51.990636    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r" event={"ID":"e6d4776d-e845-4fd5-bfab-0fd94598c75d","Type":"ContainerDied","Data":"0137e7384c8720feb37f87a9eb03921df3d0bbdc081886a04cc3516ab749d296"}
Oct 30 21:02:53.194159 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:53.194130    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:53.258456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:53.258357    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-openshift-ca\") pod \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") "
Oct 30 21:02:53.258614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:53.258540    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-xks-ca\") pod \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") "
Oct 30 21:02:53.258614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:53.258587    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d4776d-e845-4fd5-bfab-0fd94598c75d-kube-api-access-b7vxv\" (UniqueName: \"kubernetes.io/projected/e6d4776d-e845-4fd5-bfab-0fd94598c75d-kube-api-access-b7vxv\") pod \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\" (UID: \"e6d4776d-e845-4fd5-bfab-0fd94598c75d\") "
Oct 30 21:02:53.258829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:53.258798    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-openshift-ca" pod "e6d4776d-e845-4fd5-bfab-0fd94598c75d" (UID: "e6d4776d-e845-4fd5-bfab-0fd94598c75d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:53.259312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:53.259242    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.262011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:53.261976    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d4776d-e845-4fd5-bfab-0fd94598c75d-kube-api-access-b7vxv" pod "e6d4776d-e845-4fd5-bfab-0fd94598c75d" (UID: "e6d4776d-e845-4fd5-bfab-0fd94598c75d"). InnerVolumeSpecName "kube-api-access-b7vxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:53.274236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:53.274200    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-xks-ca" pod "e6d4776d-e845-4fd5-bfab-0fd94598c75d" (UID: "e6d4776d-e845-4fd5-bfab-0fd94598c75d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:53.360284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:53.360252    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6d4776d-e845-4fd5-bfab-0fd94598c75d-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:53.360284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:53.360281    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7vxv\" (UniqueName: \"kubernetes.io/projected/e6d4776d-e845-4fd5-bfab-0fd94598c75d-kube-api-access-b7vxv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:54.013959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:54.013706    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"
Oct 30 21:02:54.013959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:54.013744    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r" event={"ID":"e6d4776d-e845-4fd5-bfab-0fd94598c75d","Type":"ContainerDied","Data":"86238d99b7a2515050b85ac6bb30501d90277bf1b36630c7a296e21307a7e7f6"}
Oct 30 21:02:54.013959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:54.013801    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86238d99b7a2515050b85ac6bb30501d90277bf1b36630c7a296e21307a7e7f6"
Oct 30 21:02:55.172723 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:02:55.172681    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:02:57.448515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.448477    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"]
Oct 30 21:02:57.449352 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.449329    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e6d4776d-e845-4fd5-bfab-0fd94598c75d" containerName="revoke-keys"
Oct 30 21:02:57.449352 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.449351    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d4776d-e845-4fd5-bfab-0fd94598c75d" containerName="revoke-keys"
Oct 30 21:02:57.449510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.449495    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d4776d-e845-4fd5-bfab-0fd94598c75d" containerName="revoke-keys"
Oct 30 21:02:57.455213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.455192    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:02:57.470819 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.470794    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f64640-dockercfg-nlr7m\""
Oct 30 21:02:57.471109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.471090    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f64640\""
Oct 30 21:02:57.486198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.486174    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"]
Oct 30 21:02:57.615783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.615747    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97bsl\" (UniqueName: \"kubernetes.io/projected/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-kube-api-access-97bsl\") pod \"payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b\" (UID: \"9ac2c41a-2d51-4b93-acd2-55e9271b45ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:02:57.623314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.623283    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b\" (UID: \"9ac2c41a-2d51-4b93-acd2-55e9271b45ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:02:57.737932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.733127    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b\" (UID: \"9ac2c41a-2d51-4b93-acd2-55e9271b45ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:02:57.737932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.733489    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bsl\" (UniqueName: \"kubernetes.io/projected/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-kube-api-access-97bsl\") pod \"payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b\" (UID: \"9ac2c41a-2d51-4b93-acd2-55e9271b45ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:02:57.738515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.738477    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"]
Oct 30 21:02:57.760630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.757685    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b\" (UID: \"9ac2c41a-2d51-4b93-acd2-55e9271b45ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:02:57.768012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.767194    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"]
Oct 30 21:02:57.768012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.767315    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:02:57.768012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.767373    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bsl\" (UniqueName: \"kubernetes.io/projected/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-kube-api-access-97bsl\") pod \"payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b\" (UID: \"9ac2c41a-2d51-4b93-acd2-55e9271b45ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:02:57.794340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.794307    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:02:57.936641 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.936590    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsn9q\" (UniqueName: \"kubernetes.io/projected/0e472de9-60b5-47ae-9889-8540174d0bbf-kube-api-access-wsn9q\") pod \"payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb\" (UID: \"0e472de9-60b5-47ae-9889-8540174d0bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:02:57.936807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:57.936753    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e472de9-60b5-47ae-9889-8540174d0bbf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb\" (UID: \"0e472de9-60b5-47ae-9889-8540174d0bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:02:58.037988 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:58.037899    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsn9q\" (UniqueName: \"kubernetes.io/projected/0e472de9-60b5-47ae-9889-8540174d0bbf-kube-api-access-wsn9q\") pod \"payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb\" (UID: \"0e472de9-60b5-47ae-9889-8540174d0bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:02:58.038156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:58.038011    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e472de9-60b5-47ae-9889-8540174d0bbf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb\" (UID: \"0e472de9-60b5-47ae-9889-8540174d0bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:02:58.038704 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:58.038685    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e472de9-60b5-47ae-9889-8540174d0bbf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb\" (UID: \"0e472de9-60b5-47ae-9889-8540174d0bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:02:58.057426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:58.057389    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsn9q\" (UniqueName: \"kubernetes.io/projected/0e472de9-60b5-47ae-9889-8540174d0bbf-kube-api-access-wsn9q\") pod \"payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb\" (UID: \"0e472de9-60b5-47ae-9889-8540174d0bbf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:02:58.111662 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:58.111603    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:02:58.112408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:58.112383    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"]
Oct 30 21:02:58.418543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:58.418506    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"]
Oct 30 21:02:58.418708 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:02:58.418568    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e472de9_60b5_47ae_9889_8540174d0bbf.slice/crio-96445c54e405de232916a8e45bbc1cf6fea2cfbf378b84f624857c1ec612b311 WatchSource:0}: Error finding container 96445c54e405de232916a8e45bbc1cf6fea2cfbf378b84f624857c1ec612b311: Status 404 returned error can't find the container with id 96445c54e405de232916a8e45bbc1cf6fea2cfbf378b84f624857c1ec612b311
Oct 30 21:02:59.066073 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:59.066028    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b" event={"ID":"9ac2c41a-2d51-4b93-acd2-55e9271b45ab","Type":"ContainerStarted","Data":"f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c"}
Oct 30 21:02:59.066583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:59.066095    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:02:59.066583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:59.066117    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b" event={"ID":"9ac2c41a-2d51-4b93-acd2-55e9271b45ab","Type":"ContainerStarted","Data":"b6508ad8664b8fcabeaa4d51e7fd6705bc72b010a9175e209f89d6160f72cf58"}
Oct 30 21:02:59.068010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:59.067984    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb" event={"ID":"0e472de9-60b5-47ae-9889-8540174d0bbf","Type":"ContainerStarted","Data":"6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27"}
Oct 30 21:02:59.068147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:59.068030    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb" event={"ID":"0e472de9-60b5-47ae-9889-8540174d0bbf","Type":"ContainerStarted","Data":"96445c54e405de232916a8e45bbc1cf6fea2cfbf378b84f624857c1ec612b311"}
Oct 30 21:02:59.068147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:59.068057    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:02:59.113112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:59.113062    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b" podStartSLOduration=2.113046962 podStartE2EDuration="2.113046962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:59.093179044 +0000 UTC m=+1545.584995591" watchObservedRunningTime="2026-10-30 21:02:59.113046962 +0000 UTC m=+1545.604863501"
Oct 30 21:02:59.116077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:02:59.116038    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb" podStartSLOduration=2.116025867 podStartE2EDuration="2.116025867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:59.109840264 +0000 UTC m=+1545.601656809" watchObservedRunningTime="2026-10-30 21:02:59.116025867 +0000 UTC m=+1545.607842409"
Oct 30 21:03:00.520174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.520066    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"]
Oct 30 21:03:00.527280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.527067    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.532032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.532003    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"]
Oct 30 21:03:00.536933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.536910    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-176db5\"/\"openshift-service-ca.crt\""
Oct 30 21:03:00.537243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.537223    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-176db5\"/\"auth-test-model-176db5-kserve-self-signed-certs\""
Oct 30 21:03:00.537334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.537262    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-176db5\"/\"kube-root-ca.crt\""
Oct 30 21:03:00.537334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.537290    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-176db5\"/\"default-dockercfg-jsmrn\""
Oct 30 21:03:00.583554 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.583515    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-dshm\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.583756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.583739    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3002922a-ae83-42b9-8d51-08d606c5433f-tls-certs\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.583830 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.583785    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-model-cache\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.583921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.583840    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwlwc\" (UniqueName: \"kubernetes.io/projected/3002922a-ae83-42b9-8d51-08d606c5433f-kube-api-access-zwlwc\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.583921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.583912    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-tmp-dir\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.584030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.584006    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-home\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.684959    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwlwc\" (UniqueName: \"kubernetes.io/projected/3002922a-ae83-42b9-8d51-08d606c5433f-kube-api-access-zwlwc\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.685011    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-tmp-dir\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.685064    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-home\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.685151    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-dshm\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.685274    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3002922a-ae83-42b9-8d51-08d606c5433f-tls-certs\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.685312    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-model-cache\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.685659    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-model-cache\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.686286    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-tmp-dir\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.686459    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-home\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.691873    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3002922a-ae83-42b9-8d51-08d606c5433f-tls-certs\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.693914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.692354    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-dshm\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.695775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.695747    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwlwc\" (UniqueName: \"kubernetes.io/projected/3002922a-ae83-42b9-8d51-08d606c5433f-kube-api-access-zwlwc\") pod \"auth-test-model-176db5-kserve-764c8b88b9-bsq8v\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") " pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:00.857498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:00.857117    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:01.222516 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:03:01.221476    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3002922a_ae83_42b9_8d51_08d606c5433f.slice/crio-57dc3545c2c9466250186df944c4764666f4e6225be3565b0620221ff4b9cfb0 WatchSource:0}: Error finding container 57dc3545c2c9466250186df944c4764666f4e6225be3565b0620221ff4b9cfb0: Status 404 returned error can't find the container with id 57dc3545c2c9466250186df944c4764666f4e6225be3565b0620221ff4b9cfb0
Oct 30 21:03:01.234881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:01.224363    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"]
Oct 30 21:03:01.760821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:01.760785    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"]
Oct 30 21:03:01.764809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:01.764781    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fa48f3-mcrlj"]
Oct 30 21:03:02.111357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:02.111277    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v" event={"ID":"3002922a-ae83-42b9-8d51-08d606c5433f","Type":"ContainerStarted","Data":"c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8"}
Oct 30 21:03:02.111357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:02.111318    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v" event={"ID":"3002922a-ae83-42b9-8d51-08d606c5433f","Type":"ContainerStarted","Data":"57dc3545c2c9466250186df944c4764666f4e6225be3565b0620221ff4b9cfb0"}
Oct 30 21:03:02.134949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:02.134881    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v" podStartSLOduration=2.134846715 podStartE2EDuration="2.134846715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:02.129593192 +0000 UTC m=+1548.621409737" watchObservedRunningTime="2026-10-30 21:03:02.134846715 +0000 UTC m=+1548.626663284"
Oct 30 21:03:02.858247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:02.858208    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:03.531543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.531509    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"]
Oct 30 21:03:03.537562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.537538    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:03:03.543883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.543697    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-321ccd\""
Oct 30 21:03:03.544986 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.544961    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-321ccd-dockercfg-6vg9r\""
Oct 30 21:03:03.547902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.547879    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"]
Oct 30 21:03:03.654885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.654210    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"]
Oct 30 21:03:03.665779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.665747    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"]
Oct 30 21:03:03.665983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.665905    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:03:03.680238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.679967    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803b054b-e6d5-47ed-b254-75a8cf95d38a" path="/var/lib/kubelet/pods/803b054b-e6d5-47ed-b254-75a8cf95d38a/volumes"
Oct 30 21:03:03.721297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.690484    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd085a4e-4d27-4397-b0db-7f010b6b4cad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g\" (UID: \"cd085a4e-4d27-4397-b0db-7f010b6b4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:03:03.721297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.706160    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77v79\" (UniqueName: \"kubernetes.io/projected/cd085a4e-4d27-4397-b0db-7f010b6b4cad-kube-api-access-77v79\") pod \"payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g\" (UID: \"cd085a4e-4d27-4397-b0db-7f010b6b4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:03:03.808738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.808634    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd085a4e-4d27-4397-b0db-7f010b6b4cad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g\" (UID: \"cd085a4e-4d27-4397-b0db-7f010b6b4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:03:03.808738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.808704    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnsx\" (UniqueName: \"kubernetes.io/projected/224bb67a-9e62-48c5-b703-a8de47cf7f92-kube-api-access-5qnsx\") pod \"payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp\" (UID: \"224bb67a-9e62-48c5-b703-a8de47cf7f92\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:03:03.810277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.808757    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224bb67a-9e62-48c5-b703-a8de47cf7f92-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp\" (UID: \"224bb67a-9e62-48c5-b703-a8de47cf7f92\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:03:03.810277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.808806    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77v79\" (UniqueName: \"kubernetes.io/projected/cd085a4e-4d27-4397-b0db-7f010b6b4cad-kube-api-access-77v79\") pod \"payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g\" (UID: \"cd085a4e-4d27-4397-b0db-7f010b6b4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:03:03.810277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.810192    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd085a4e-4d27-4397-b0db-7f010b6b4cad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g\" (UID: \"cd085a4e-4d27-4397-b0db-7f010b6b4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:03:03.830427 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.830393    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77v79\" (UniqueName: \"kubernetes.io/projected/cd085a4e-4d27-4397-b0db-7f010b6b4cad-kube-api-access-77v79\") pod \"payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g\" (UID: \"cd085a4e-4d27-4397-b0db-7f010b6b4cad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:03:03.865934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.865902    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:03:03.910781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.910729    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnsx\" (UniqueName: \"kubernetes.io/projected/224bb67a-9e62-48c5-b703-a8de47cf7f92-kube-api-access-5qnsx\") pod \"payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp\" (UID: \"224bb67a-9e62-48c5-b703-a8de47cf7f92\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:03:03.911112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.910804    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224bb67a-9e62-48c5-b703-a8de47cf7f92-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp\" (UID: \"224bb67a-9e62-48c5-b703-a8de47cf7f92\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:03:03.916400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.916372    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224bb67a-9e62-48c5-b703-a8de47cf7f92-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp\" (UID: \"224bb67a-9e62-48c5-b703-a8de47cf7f92\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:03:03.926890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:03.926575    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnsx\" (UniqueName: \"kubernetes.io/projected/224bb67a-9e62-48c5-b703-a8de47cf7f92-kube-api-access-5qnsx\") pod \"payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp\" (UID: \"224bb67a-9e62-48c5-b703-a8de47cf7f92\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:03:04.041160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:04.033595    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:03:04.142130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:04.142062    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"]
Oct 30 21:03:04.152486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:04.152414    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b46fb7-84zqj"]
Oct 30 21:03:04.303340 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:03:04.301924    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd085a4e_4d27_4397_b0db_7f010b6b4cad.slice/crio-79f6b4be7b35a70f3c5523f0269f529831b77d3054671e57b85cd2eea45acd28 WatchSource:0}: Error finding container 79f6b4be7b35a70f3c5523f0269f529831b77d3054671e57b85cd2eea45acd28: Status 404 returned error can't find the container with id 79f6b4be7b35a70f3c5523f0269f529831b77d3054671e57b85cd2eea45acd28
Oct 30 21:03:04.314216 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:04.308957    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"]
Oct 30 21:03:04.428582 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:03:04.428541    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224bb67a_9e62_48c5_b703_a8de47cf7f92.slice/crio-f12e48f198417034a8b068f9c170b33915adb0cfb746430e219456c2243dd6b3 WatchSource:0}: Error finding container f12e48f198417034a8b068f9c170b33915adb0cfb746430e219456c2243dd6b3: Status 404 returned error can't find the container with id f12e48f198417034a8b068f9c170b33915adb0cfb746430e219456c2243dd6b3
Oct 30 21:03:04.429417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:04.429391    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"]
Oct 30 21:03:05.158662 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:05.158618    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g" event={"ID":"cd085a4e-4d27-4397-b0db-7f010b6b4cad","Type":"ContainerStarted","Data":"c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020"}
Oct 30 21:03:05.159167 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:05.158687    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g" event={"ID":"cd085a4e-4d27-4397-b0db-7f010b6b4cad","Type":"ContainerStarted","Data":"79f6b4be7b35a70f3c5523f0269f529831b77d3054671e57b85cd2eea45acd28"}
Oct 30 21:03:05.159167 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:05.158716    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:03:05.161150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:05.161119    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp" event={"ID":"224bb67a-9e62-48c5-b703-a8de47cf7f92","Type":"ContainerStarted","Data":"5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18"}
Oct 30 21:03:05.161318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:05.161176    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp" event={"ID":"224bb67a-9e62-48c5-b703-a8de47cf7f92","Type":"ContainerStarted","Data":"f12e48f198417034a8b068f9c170b33915adb0cfb746430e219456c2243dd6b3"}
Oct 30 21:03:05.161318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:05.161204    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:03:05.178359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:05.178298    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g" podStartSLOduration=2.178276487 podStartE2EDuration="2.178276487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:05.175376558 +0000 UTC m=+1551.667193104" watchObservedRunningTime="2026-10-30 21:03:05.178276487 +0000 UTC m=+1551.670093036"
Oct 30 21:03:05.198631 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:05.198569    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp" podStartSLOduration=2.198550013 podStartE2EDuration="2.198550013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:05.193041191 +0000 UTC m=+1551.684857735" watchObservedRunningTime="2026-10-30 21:03:05.198550013 +0000 UTC m=+1551.690366558"
Oct 30 21:03:05.663300 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:05.663259    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9127c3f-272b-4144-aef6-234d64f12339" path="/var/lib/kubelet/pods/f9127c3f-272b-4144-aef6-234d64f12339/volumes"
Oct 30 21:03:09.040886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:09.040836    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:03:10.081252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:10.081222    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:03:10.081788 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:10.081770    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:03:10.857895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:10.857844    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:10.875655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:10.875617    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:11.297819 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:11.297788    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:03:15.339166 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:15.339131    4810 scope.go:122] "RemoveContainer" containerID="88875e12238e72885e01ce0cc0b544791de4fbd3e11d7a9a9b6b4bb2e07afb75"
Oct 30 21:03:15.351755 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:15.351734    4810 scope.go:122] "RemoveContainer" containerID="cf3989e9f6497b7be22240cf5ac4f34169aea3bf9999000003ee07e113a550d8"
Oct 30 21:03:15.361955 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:15.361931    4810 scope.go:122] "RemoveContainer" containerID="ff2171dc0953b2c12fe9fe91ba98cd8d8a34d2b8a21096eb6953b586162feb84"
Oct 30 21:03:15.371406 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:15.371378    4810 scope.go:122] "RemoveContainer" containerID="f3eb5d51c9e0b828e3bc12df54a53aa2001c3de18d2450d157881c8910157367"
Oct 30 21:03:16.194896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:16.194009    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:03:16.225878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:16.222071    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:03:17.086875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.086831    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"]
Oct 30 21:03:17.095007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.094975    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.103345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.103318    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"]
Oct 30 21:03:17.110150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.109810    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-74f9f4\"/\"openshift-service-ca.crt\""
Oct 30 21:03:17.110150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.109915    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-74f9f4\"/\"kube-root-ca.crt\""
Oct 30 21:03:17.110150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.110050    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-74f9f4\"/\"auth-test-model-74f9f4-kserve-self-signed-certs\""
Oct 30 21:03:17.110712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.110692    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-74f9f4\"/\"default-dockercfg-lkw59\""
Oct 30 21:03:17.151998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.151960    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a353a44-d648-4ada-9234-2314b56e4eaa-tls-certs\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.152365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.152339    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-tmp-dir\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.152466 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.152383    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-model-cache\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.152557 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.152536    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-home\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.152695 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.152680    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-dshm\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.152755 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.152736    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xt9q\" (UniqueName: \"kubernetes.io/projected/4a353a44-d648-4ada-9234-2314b56e4eaa-kube-api-access-9xt9q\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.254040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.253998    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-tmp-dir\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.254281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.254053    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-model-cache\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.254281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.254184    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-home\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.254412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.254336    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-dshm\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.254412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.254391    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xt9q\" (UniqueName: \"kubernetes.io/projected/4a353a44-d648-4ada-9234-2314b56e4eaa-kube-api-access-9xt9q\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.254518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.254458    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-tmp-dir\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.254580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.254521    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-model-cache\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.254580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.254547    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-home\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.254816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.254462    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a353a44-d648-4ada-9234-2314b56e4eaa-tls-certs\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.256621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.256595    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-dshm\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.256902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.256853    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a353a44-d648-4ada-9234-2314b56e4eaa-tls-certs\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.265892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.265851    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xt9q\" (UniqueName: \"kubernetes.io/projected/4a353a44-d648-4ada-9234-2314b56e4eaa-kube-api-access-9xt9q\") pod \"auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") " pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.418150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.418111    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:17.760022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:17.759950    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"]
Oct 30 21:03:18.479767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:18.479722    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw" event={"ID":"4a353a44-d648-4ada-9234-2314b56e4eaa","Type":"ContainerStarted","Data":"fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449"}
Oct 30 21:03:18.480262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:18.479787    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw" event={"ID":"4a353a44-d648-4ada-9234-2314b56e4eaa","Type":"ContainerStarted","Data":"3e69465885068cbe5e87210691d44101402dbf214729cface7a157019a3458b3"}
Oct 30 21:03:18.501485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:18.501410    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw" podStartSLOduration=1.501390776 podStartE2EDuration="1.501390776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:18.499354184 +0000 UTC m=+1564.991170733" watchObservedRunningTime="2026-10-30 21:03:18.501390776 +0000 UTC m=+1564.993207321"
Oct 30 21:03:19.419503 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:19.419473    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:25.158942 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:03:25.158816    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:03:27.419797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:27.419752    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:27.433980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:27.433952    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:27.641322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:27.641295    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:03:31.394007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.393468    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"]
Oct 30 21:03:31.418323 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.416350    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.421147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.421046    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"]
Oct 30 21:03:31.423385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.423362    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8fa15b\"/\"default-dockercfg-zkksn\""
Oct 30 21:03:31.426486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.426463    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8fa15b\"/\"kube-root-ca.crt\""
Oct 30 21:03:31.426729 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.426709    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8fa15b\"/\"openshift-service-ca.crt\""
Oct 30 21:03:31.427103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.427055    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8fa15b\"/\"auth-test-model-8fa15b-kserve-self-signed-certs\""
Oct 30 21:03:31.556595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.556557    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tls-certs\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.557237 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.557151    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-model-cache\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.558624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.558597    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tmp-dir\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.558829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.558809    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-home\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.559021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.558998    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsgd7\" (UniqueName: \"kubernetes.io/projected/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-kube-api-access-dsgd7\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.559222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.559207    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-dshm\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.669570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.669481    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-model-cache\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.669994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.669972    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tmp-dir\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.670236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.670208    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-home\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.670396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.670382    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgd7\" (UniqueName: \"kubernetes.io/projected/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-kube-api-access-dsgd7\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.670572 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.670553    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-dshm\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.670758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.670742    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tls-certs\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.680889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.671781    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-home\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.681359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.681334    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-model-cache\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.681716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.681697    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tmp-dir\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.721884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.708035    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-dshm\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.745270 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.745243    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgd7\" (UniqueName: \"kubernetes.io/projected/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-kube-api-access-dsgd7\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:31.752384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:31.752348    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tls-certs\") pod \"auth-test-model-8fa15b-kserve-856f6f8565-ctcwg\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") " pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:32.035741 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:32.035649    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:32.347061 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:03:32.347029    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fbe7c00_2747_4ca6_ba6c_52a5f9a9a709.slice/crio-72ce24f7ce68d2493a561955280ead04e2e24e53c4169a9df0ece7dd7782e182 WatchSource:0}: Error finding container 72ce24f7ce68d2493a561955280ead04e2e24e53c4169a9df0ece7dd7782e182: Status 404 returned error can't find the container with id 72ce24f7ce68d2493a561955280ead04e2e24e53c4169a9df0ece7dd7782e182
Oct 30 21:03:32.347560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:32.347540    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"]
Oct 30 21:03:32.762910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:32.761261    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg" event={"ID":"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709","Type":"ContainerStarted","Data":"f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8"}
Oct 30 21:03:32.762910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:32.761313    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg" event={"ID":"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709","Type":"ContainerStarted","Data":"72ce24f7ce68d2493a561955280ead04e2e24e53c4169a9df0ece7dd7782e182"}
Oct 30 21:03:32.780108 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:32.780034    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg" podStartSLOduration=1.780016321 podStartE2EDuration="1.780016321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:32.779635724 +0000 UTC m=+1579.271452270" watchObservedRunningTime="2026-10-30 21:03:32.780016321 +0000 UTC m=+1579.271832869"
Oct 30 21:03:33.044054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:33.043949    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:42.038760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.038722    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:42.051717 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.051688    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:42.393607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.393575    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"]
Oct 30 21:03:42.393797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.393767    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy" containerID="cri-o://b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e" gracePeriod=30
Oct 30 21:03:42.745925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.745820    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"]
Oct 30 21:03:42.751784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.751757    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.762316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.762291    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"]
Oct 30 21:03:42.763920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.763898    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e53c37\"/\"openshift-service-ca.crt\""
Oct 30 21:03:42.764078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.763903    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e53c37\"/\"default-dockercfg-gpwpn\""
Oct 30 21:03:42.764142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.763972    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e53c37\"/\"auth-test-model-e53c37-kserve-self-signed-certs\""
Oct 30 21:03:42.764142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.763994    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e53c37\"/\"kube-root-ca.crt\""
Oct 30 21:03:42.852807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.852773    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-model-cache\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.852992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.852839    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-dshm\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.852992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.852974    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39858eb-242d-4afe-979a-74216af0dce4-tls-certs\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.853072 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.853014    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhhgv\" (UniqueName: \"kubernetes.io/projected/c39858eb-242d-4afe-979a-74216af0dce4-kube-api-access-fhhgv\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.853162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.853145    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-tmp-dir\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.853206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.853170    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-home\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.884911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.884880    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:03:42.955227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.955197    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-tmp-dir\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.955417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.955236    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-home\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.955417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.955361    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-model-cache\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.955537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.955431    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-dshm\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.955537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.955486    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39858eb-242d-4afe-979a-74216af0dce4-tls-certs\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.955640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.955536    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhhgv\" (UniqueName: \"kubernetes.io/projected/c39858eb-242d-4afe-979a-74216af0dce4-kube-api-access-fhhgv\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.956634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.956601    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-tmp-dir\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.956764 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.956683    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-model-cache\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.956972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.956949    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-home\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.958904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.958813    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39858eb-242d-4afe-979a-74216af0dce4-tls-certs\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.959227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.959203    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-dshm\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:42.965925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:42.965882    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhhgv\" (UniqueName: \"kubernetes.io/projected/c39858eb-242d-4afe-979a-74216af0dce4-kube-api-access-fhhgv\") pod \"auth-test-model-e53c37-kserve-7897c66b9-mkc64\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") " pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:43.099027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:43.081156    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:43.466997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:43.466967    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"]
Oct 30 21:03:43.467503 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:03:43.467474    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc39858eb_242d_4afe_979a_74216af0dce4.slice/crio-ebe2c8c6d9a2ae8c3a466d65681cbf67ae1debd229c153404757be11fee7b8bd WatchSource:0}: Error finding container ebe2c8c6d9a2ae8c3a466d65681cbf67ae1debd229c153404757be11fee7b8bd: Status 404 returned error can't find the container with id ebe2c8c6d9a2ae8c3a466d65681cbf67ae1debd229c153404757be11fee7b8bd
Oct 30 21:03:43.882759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:43.882722    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64" event={"ID":"c39858eb-242d-4afe-979a-74216af0dce4","Type":"ContainerStarted","Data":"aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169"}
Oct 30 21:03:43.882990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:43.882776    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64" event={"ID":"c39858eb-242d-4afe-979a-74216af0dce4","Type":"ContainerStarted","Data":"ebe2c8c6d9a2ae8c3a466d65681cbf67ae1debd229c153404757be11fee7b8bd"}
Oct 30 21:03:43.904839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:43.904791    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64" podStartSLOduration=1.904776178 podStartE2EDuration="1.904776178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:43.899237052 +0000 UTC m=+1590.391053597" watchObservedRunningTime="2026-10-30 21:03:43.904776178 +0000 UTC m=+1590.396592723"
Oct 30 21:03:44.088365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:44.087943    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:47.595028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:47.594986    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:03:47.900310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:47.900277    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 21:03:47.938410 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:47.938373    4810 generic.go:356] "Generic (PLEG): container finished" podID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerID="b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e" exitCode=0
Oct 30 21:03:47.938600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:47.938434    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" event={"ID":"4e6d65a3-e831-4dd3-bc81-1121b26717c2","Type":"ContainerDied","Data":"b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e"}
Oct 30 21:03:47.938600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:47.938485    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n" event={"ID":"4e6d65a3-e831-4dd3-bc81-1121b26717c2","Type":"ContainerDied","Data":"24ed3932357c0a8edc93f9cf0c81565a53171b4f097377113c9979ad1ead738c"}
Oct 30 21:03:47.938600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:47.938513    4810 scope.go:122] "RemoveContainer" containerID="b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e"
Oct 30 21:03:47.938779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:47.938675    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"
Oct 30 21:03:47.953771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:47.953751    4810 scope.go:122] "RemoveContainer" containerID="b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e"
Oct 30 21:03:47.954159 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:03:47.954133    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e\": container with ID starting with b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e not found: ID does not exist" containerID="b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e"
Oct 30 21:03:47.954244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:47.954173    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e"} err="failed to get container status \"b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e\": rpc error: code = NotFound desc = could not find container \"b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e\": container with ID starting with b5a30af7eda023802abdf0b0265eb386a88d3a8e134b57e289a6bc69c742391e not found: ID does not exist"
Oct 30 21:03:48.065487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.065450    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-socket\") pod \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") "
Oct 30 21:03:48.065661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.065538    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-envoy\") pod \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") "
Oct 30 21:03:48.065661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.065587    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istiod-ca-cert\") pod \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") "
Oct 30 21:03:48.065754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.065688    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-certs\") pod \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") "
Oct 30 21:03:48.065754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.065717    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-data\") pod \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") "
Oct 30 21:03:48.065869 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.065757    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-kube-api-access-cmt5t\" (UniqueName: \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-kube-api-access-cmt5t\") pod \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") "
Oct 30 21:03:48.065869 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.065791    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-socket" pod "4e6d65a3-e831-4dd3-bc81-1121b26717c2" (UID: "4e6d65a3-e831-4dd3-bc81-1121b26717c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:48.065869 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.065849    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-token\" (UniqueName: \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-token\") pod \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") "
Oct 30 21:03:48.066035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.065930    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-credential-socket\") pod \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") "
Oct 30 21:03:48.066035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.065966    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-podinfo\") pod \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\" (UID: \"4e6d65a3-e831-4dd3-bc81-1121b26717c2\") "
Oct 30 21:03:48.066148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.066064    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-certs" pod "4e6d65a3-e831-4dd3-bc81-1121b26717c2" (UID: "4e6d65a3-e831-4dd3-bc81-1121b26717c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:48.066209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.066160    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-data" pod "4e6d65a3-e831-4dd3-bc81-1121b26717c2" (UID: "4e6d65a3-e831-4dd3-bc81-1121b26717c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:48.066209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.066190    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istiod-ca-cert" pod "4e6d65a3-e831-4dd3-bc81-1121b26717c2" (UID: "4e6d65a3-e831-4dd3-bc81-1121b26717c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:48.066536 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.066500    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-credential-socket" pod "4e6d65a3-e831-4dd3-bc81-1121b26717c2" (UID: "4e6d65a3-e831-4dd3-bc81-1121b26717c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:48.067038 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.067012    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:48.067182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.067043    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:48.067182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.067058    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:48.067182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.067073    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:48.067182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.067087    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:48.068137 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.068114    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-envoy" pod "4e6d65a3-e831-4dd3-bc81-1121b26717c2" (UID: "4e6d65a3-e831-4dd3-bc81-1121b26717c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:48.068283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.068258    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-token" pod "4e6d65a3-e831-4dd3-bc81-1121b26717c2" (UID: "4e6d65a3-e831-4dd3-bc81-1121b26717c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:48.068570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.068548    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-podinfo" pod "4e6d65a3-e831-4dd3-bc81-1121b26717c2" (UID: "4e6d65a3-e831-4dd3-bc81-1121b26717c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:48.068671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.068609    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-kube-api-access-cmt5t" pod "4e6d65a3-e831-4dd3-bc81-1121b26717c2" (UID: "4e6d65a3-e831-4dd3-bc81-1121b26717c2"). InnerVolumeSpecName "kube-api-access-cmt5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:48.168373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.168341    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:48.168373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.168370    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmt5t\" (UniqueName: \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-kube-api-access-cmt5t\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:48.168373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.168381    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:48.168599 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.168391    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6d65a3-e831-4dd3-bc81-1121b26717c2-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:48.281333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.281292    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"]
Oct 30 21:03:48.291476 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:48.291446    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-bkg5n"]
Oct 30 21:03:49.664281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:49.664244    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" path="/var/lib/kubelet/pods/4e6d65a3-e831-4dd3-bc81-1121b26717c2/volumes"
Oct 30 21:03:53.082177 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:53.082130    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:53.100365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:53.100342    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:54.014542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:54.014502    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:03:55.156805 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:03:55.156771    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:03:57.256595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.256558    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"]
Oct 30 21:03:57.257278 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.257262    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy"
Oct 30 21:03:57.257324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.257278    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy"
Oct 30 21:03:57.257425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.257416    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6d65a3-e831-4dd3-bc81-1121b26717c2" containerName="istio-proxy"
Oct 30 21:03:57.262773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.262749    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.271201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.271177    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"]
Oct 30 21:03:57.324818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.324790    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:03:57.324981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.324786    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:03:57.330775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.330744    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl8fv\" (UniqueName: \"kubernetes.io/projected/bf559d4c-415a-44bc-959e-be19cea9ba8b-kube-api-access-hl8fv\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.330898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.330827    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.330949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.330898    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.432483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.432449    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl8fv\" (UniqueName: \"kubernetes.io/projected/bf559d4c-415a-44bc-959e-be19cea9ba8b-kube-api-access-hl8fv\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.432675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.432516    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.432675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.432555    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.433315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.433289    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.434972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.434947    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.441592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.441571    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl8fv\" (UniqueName: \"kubernetes.io/projected/bf559d4c-415a-44bc-959e-be19cea9ba8b-kube-api-access-hl8fv\") pod \"maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.601915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.601838    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:03:57.893541 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:57.893507    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"]
Oct 30 21:03:57.894089 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:03:57.894061    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf559d4c_415a_44bc_959e_be19cea9ba8b.slice/crio-fd798d5533aba208ed8f7cafd1975f1597b210a379c9c593f893b21cb9cb955b WatchSource:0}: Error finding container fd798d5533aba208ed8f7cafd1975f1597b210a379c9c593f893b21cb9cb955b: Status 404 returned error can't find the container with id fd798d5533aba208ed8f7cafd1975f1597b210a379c9c593f893b21cb9cb955b
Oct 30 21:03:58.049584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.049553    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws" event={"ID":"bf559d4c-415a-44bc-959e-be19cea9ba8b","Type":"ContainerStarted","Data":"fd798d5533aba208ed8f7cafd1975f1597b210a379c9c593f893b21cb9cb955b"}
Oct 30 21:03:58.432795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.432759    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"]
Oct 30 21:03:58.450921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.450889    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"]
Oct 30 21:03:58.451088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.451012    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.455778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.455748    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f64640\"/\"openshift-service-ca.crt\""
Oct 30 21:03:58.455944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.455842    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f64640\"/\"kube-root-ca.crt\""
Oct 30 21:03:58.455944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.455852    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f64640\"/\"auth-test-model-f64640-kserve-self-signed-certs\""
Oct 30 21:03:58.455944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.455896    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f64640\"/\"default-dockercfg-nnxd9\""
Oct 30 21:03:58.551981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.551893    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tmp-dir\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.551981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.551935    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc7kn\" (UniqueName: \"kubernetes.io/projected/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-kube-api-access-kc7kn\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.552227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.552142    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-dshm\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.552227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.552188    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-home\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.552363 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.552349    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tls-certs\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.552419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.552385    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-model-cache\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.661750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.655710    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-home\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.661750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.655830    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tls-certs\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.661750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.655880    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-model-cache\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.661750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.655984    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tmp-dir\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.661750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.656012    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc7kn\" (UniqueName: \"kubernetes.io/projected/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-kube-api-access-kc7kn\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.661750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.656147    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-dshm\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.661750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.656511    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-model-cache\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.661750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.657677    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tmp-dir\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.661750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.658249    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-home\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.677907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.673128    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-dshm\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.678700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.678644    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tls-certs\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.681533 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.681484    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc7kn\" (UniqueName: \"kubernetes.io/projected/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-kube-api-access-kc7kn\") pod \"auth-test-model-f64640-kserve-b76d57b85-z4j52\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") " pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:58.787469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:58.787434    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:03:59.059042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:59.059005    4810 generic.go:356] "Generic (PLEG): container finished" podID="bf559d4c-415a-44bc-959e-be19cea9ba8b" containerID="7d7c2f36f367f3bc9448aae79030ddfc91a43d6490b4715a1c962d4657ea1900" exitCode=0
Oct 30 21:03:59.059182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:59.059126    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws" event={"ID":"bf559d4c-415a-44bc-959e-be19cea9ba8b","Type":"ContainerDied","Data":"7d7c2f36f367f3bc9448aae79030ddfc91a43d6490b4715a1c962d4657ea1900"}
Oct 30 21:03:59.107362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:03:59.107335    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"]
Oct 30 21:03:59.110425 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:03:59.110397    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd59cf7da_1bf1_4b55_9039_1ba1fcd18ee8.slice/crio-1f807ed4d21562f6a4e6cc5648b995e6d4de86e70cf68f51d4f58dbf1fee8426 WatchSource:0}: Error finding container 1f807ed4d21562f6a4e6cc5648b995e6d4de86e70cf68f51d4f58dbf1fee8426: Status 404 returned error can't find the container with id 1f807ed4d21562f6a4e6cc5648b995e6d4de86e70cf68f51d4f58dbf1fee8426
Oct 30 21:04:00.069899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.069845    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52" event={"ID":"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8","Type":"ContainerStarted","Data":"4e940f41620c92d484cebca5df5d95fd1ff1d240844cc738819b9995d06fcf21"}
Oct 30 21:04:00.070324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.069908    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52" event={"ID":"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8","Type":"ContainerStarted","Data":"1f807ed4d21562f6a4e6cc5648b995e6d4de86e70cf68f51d4f58dbf1fee8426"}
Oct 30 21:04:00.333120 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.333006    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:04:00.356975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.356901    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52" podStartSLOduration=2.3568480960000002 podStartE2EDuration="2.356848096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:00.093085352 +0000 UTC m=+1606.584901903" watchObservedRunningTime="2026-10-30 21:04:00.356848096 +0000 UTC m=+1606.848664643"
Oct 30 21:04:00.425076 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.425045    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-xks-ca\") pod \"bf559d4c-415a-44bc-959e-be19cea9ba8b\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") "
Oct 30 21:04:00.425264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.425091    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf559d4c-415a-44bc-959e-be19cea9ba8b-kube-api-access-hl8fv\" (UniqueName: \"kubernetes.io/projected/bf559d4c-415a-44bc-959e-be19cea9ba8b-kube-api-access-hl8fv\") pod \"bf559d4c-415a-44bc-959e-be19cea9ba8b\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") "
Oct 30 21:04:00.425324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.425260    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-openshift-ca\") pod \"bf559d4c-415a-44bc-959e-be19cea9ba8b\" (UID: \"bf559d4c-415a-44bc-959e-be19cea9ba8b\") "
Oct 30 21:04:00.425618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.425591    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-openshift-ca" pod "bf559d4c-415a-44bc-959e-be19cea9ba8b" (UID: "bf559d4c-415a-44bc-959e-be19cea9ba8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:00.427250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.427226    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf559d4c-415a-44bc-959e-be19cea9ba8b-kube-api-access-hl8fv" pod "bf559d4c-415a-44bc-959e-be19cea9ba8b" (UID: "bf559d4c-415a-44bc-959e-be19cea9ba8b"). InnerVolumeSpecName "kube-api-access-hl8fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:00.427355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.427315    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-xks-ca" pod "bf559d4c-415a-44bc-959e-be19cea9ba8b" (UID: "bf559d4c-415a-44bc-959e-be19cea9ba8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:00.526611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.526580    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:00.526611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.526612    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf559d4c-415a-44bc-959e-be19cea9ba8b-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:00.526852 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.526625    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl8fv\" (UniqueName: \"kubernetes.io/projected/bf559d4c-415a-44bc-959e-be19cea9ba8b-kube-api-access-hl8fv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:04:00.788556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:00.788519    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:04:01.092491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:01.092404    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"
Oct 30 21:04:01.092491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:01.092429    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws" event={"ID":"bf559d4c-415a-44bc-959e-be19cea9ba8b","Type":"ContainerDied","Data":"fd798d5533aba208ed8f7cafd1975f1597b210a379c9c593f893b21cb9cb955b"}
Oct 30 21:04:01.093011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:01.092492    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd798d5533aba208ed8f7cafd1975f1597b210a379c9c593f893b21cb9cb955b"
Oct 30 21:04:07.741378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.741341    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"]
Oct 30 21:04:07.742150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.742129    4810 cpu_manager.go:414] "containerMap: removing container" podUID="bf559d4c-415a-44bc-959e-be19cea9ba8b" containerName="revoke-keys"
Oct 30 21:04:07.742150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.742151    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf559d4c-415a-44bc-959e-be19cea9ba8b" containerName="revoke-keys"
Oct 30 21:04:07.742296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.742282    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf559d4c-415a-44bc-959e-be19cea9ba8b" containerName="revoke-keys"
Oct 30 21:04:07.746793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.746768    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.748335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.748307    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"]
Oct 30 21:04:07.754087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.754063    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-321ccd\"/\"default-dockercfg-wrn68\""
Oct 30 21:04:07.754226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.754188    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-321ccd\"/\"auth-test-model-321ccd-kserve-self-signed-certs\""
Oct 30 21:04:07.754728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.754710    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-321ccd\"/\"openshift-service-ca.crt\""
Oct 30 21:04:07.754957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.754941    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-321ccd\"/\"kube-root-ca.crt\""
Oct 30 21:04:07.838253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.838215    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6393d03c-e423-4037-9aef-88430e0674d1-tls-certs\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.838443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.838264    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-dshm\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.838509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.838475    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9sbc\" (UniqueName: \"kubernetes.io/projected/6393d03c-e423-4037-9aef-88430e0674d1-kube-api-access-t9sbc\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.838565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.838538    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-model-cache\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.838619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.838584    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-tmp-dir\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.838671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.838618    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-home\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.948876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.948515    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-tmp-dir\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.948876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.948593    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-home\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.948876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.948749    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6393d03c-e423-4037-9aef-88430e0674d1-tls-certs\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.948876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.948808    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-dshm\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.949223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.948931    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9sbc\" (UniqueName: \"kubernetes.io/projected/6393d03c-e423-4037-9aef-88430e0674d1-kube-api-access-t9sbc\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.949223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.949002    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-model-cache\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.959878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.959478    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-tmp-dir\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.959878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.959836    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-home\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.968808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.960458    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-model-cache\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.968808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.964330    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6393d03c-e423-4037-9aef-88430e0674d1-tls-certs\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.972687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.971121    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9sbc\" (UniqueName: \"kubernetes.io/projected/6393d03c-e423-4037-9aef-88430e0674d1-kube-api-access-t9sbc\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:07.994013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:07.987616    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-dshm\") pod \"auth-test-model-321ccd-kserve-549467c589-2nncj\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") " pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:08.067924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:08.067892    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:08.384707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:08.384673    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"]
Oct 30 21:04:08.385817 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:04:08.385791    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6393d03c_e423_4037_9aef_88430e0674d1.slice/crio-712d86cd9a4f8e04aa426f6187721bdd70ab3b2a1ad486534f03a0800476eb68 WatchSource:0}: Error finding container 712d86cd9a4f8e04aa426f6187721bdd70ab3b2a1ad486534f03a0800476eb68: Status 404 returned error can't find the container with id 712d86cd9a4f8e04aa426f6187721bdd70ab3b2a1ad486534f03a0800476eb68
Oct 30 21:04:08.788441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:08.788350    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:04:08.808299 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:08.808274    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:04:09.220399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:09.220359    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj" event={"ID":"6393d03c-e423-4037-9aef-88430e0674d1","Type":"ContainerStarted","Data":"a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653"}
Oct 30 21:04:09.220724 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:09.220694    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj" event={"ID":"6393d03c-e423-4037-9aef-88430e0674d1","Type":"ContainerStarted","Data":"712d86cd9a4f8e04aa426f6187721bdd70ab3b2a1ad486534f03a0800476eb68"}
Oct 30 21:04:09.233510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:09.233482    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:04:09.238640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:09.238577    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj" podStartSLOduration=2.238561724 podStartE2EDuration="2.238561724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:09.236692293 +0000 UTC m=+1615.728508843" watchObservedRunningTime="2026-10-30 21:04:09.238561724 +0000 UTC m=+1615.730378269"
Oct 30 21:04:10.077102 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:10.077065    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:10.551899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:10.551521    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"]
Oct 30 21:04:10.559821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:10.558175    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7c7768-lj5ws"]
Oct 30 21:04:11.664624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:11.664587    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf559d4c-415a-44bc-959e-be19cea9ba8b" path="/var/lib/kubelet/pods/bf559d4c-415a-44bc-959e-be19cea9ba8b/volumes"
Oct 30 21:04:15.425779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:15.425743    4810 scope.go:122] "RemoveContainer" containerID="3bf5100589654615c3e7aba904f987cc45bd8ec810a1dd9444011d28e1dfe7a1"
Oct 30 21:04:15.442734 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:15.442673    4810 scope.go:122] "RemoveContainer" containerID="6eb572b9899b5ef849752b5d915c851a19803f303ba20ff011ef719b64a5ca97"
Oct 30 21:04:15.456454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:15.456423    4810 scope.go:122] "RemoveContainer" containerID="ab5861eaa525bf984dd3979c4e45ca325344bab2cd05b5f60d30358cd74ac5c5"
Oct 30 21:04:15.468255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:15.468233    4810 scope.go:122] "RemoveContainer" containerID="713033b722894b772bc1435544e322aedf3399293544032266289ca5e3dfddb3"
Oct 30 21:04:15.479578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:15.479549    4810 scope.go:122] "RemoveContainer" containerID="45a0b36f977abf47b595d9a378b8c0b84f1b64115be8a618ef8c9843ce5667e9"
Oct 30 21:04:18.077880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:18.077186    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:18.105634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:18.105597    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:18.334108 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:18.334027    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:04:25.157350 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:04:25.157314    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:04:33.353286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:33.353249    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:04:43.827818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.827783    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"]
Oct 30 21:04:43.833532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.833507    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:43.841509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.841167    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-a50675\"/\"kube-root-ca.crt\""
Oct 30 21:04:43.841509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.841438    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-a50675\"/\"default-dockercfg-lrr2c\""
Oct 30 21:04:43.841509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.841484    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-a50675\"/\"openshift-service-ca.crt\""
Oct 30 21:04:43.841763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.841612    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-a50675\"/\"auth-test-model-a50675-kserve-self-signed-certs\""
Oct 30 21:04:43.848706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.848681    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"]
Oct 30 21:04:43.966344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.966300    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtm76\" (UniqueName: \"kubernetes.io/projected/52e8d99c-d846-443a-a36b-98bb6e769324-kube-api-access-qtm76\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:43.966551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.966351    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-model-cache\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:43.966551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.966461    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e8d99c-d846-443a-a36b-98bb6e769324-tls-certs\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:43.966676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.966595    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-dshm\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:43.966676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.966646    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-home\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:43.966792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:43.966715    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-tmp-dir\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.068983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.068828    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtm76\" (UniqueName: \"kubernetes.io/projected/52e8d99c-d846-443a-a36b-98bb6e769324-kube-api-access-qtm76\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.068983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.068903    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-model-cache\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.068983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.068978    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e8d99c-d846-443a-a36b-98bb6e769324-tls-certs\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.069419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.069100    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-dshm\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.069419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.069168    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-home\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.069419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.069209    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-tmp-dir\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.070062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.069667    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-tmp-dir\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.070881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.070264    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-model-cache\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.072891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.071545    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-home\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.097923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.097613    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtm76\" (UniqueName: \"kubernetes.io/projected/52e8d99c-d846-443a-a36b-98bb6e769324-kube-api-access-qtm76\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.097923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.097831    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-dshm\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.101954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.098889    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e8d99c-d846-443a-a36b-98bb6e769324-tls-certs\") pod \"auth-test-model-a50675-kserve-78ffc6568f-rvt57\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") " pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.157644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.157610    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:44.487432 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:04:44.487399    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52e8d99c_d846_443a_a36b_98bb6e769324.slice/crio-cd30bcc68dbe878050d297fe07024c3b30c11c8172b7862291cc3f714af4f1c7 WatchSource:0}: Error finding container cd30bcc68dbe878050d297fe07024c3b30c11c8172b7862291cc3f714af4f1c7: Status 404 returned error can't find the container with id cd30bcc68dbe878050d297fe07024c3b30c11c8172b7862291cc3f714af4f1c7
Oct 30 21:04:44.488026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.488002    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"]
Oct 30 21:04:44.636911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.636414    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57" event={"ID":"52e8d99c-d846-443a-a36b-98bb6e769324","Type":"ContainerStarted","Data":"bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b"}
Oct 30 21:04:44.636911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.636468    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57" event={"ID":"52e8d99c-d846-443a-a36b-98bb6e769324","Type":"ContainerStarted","Data":"cd30bcc68dbe878050d297fe07024c3b30c11c8172b7862291cc3f714af4f1c7"}
Oct 30 21:04:44.658027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:44.657975    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57" podStartSLOduration=1.657961904 podStartE2EDuration="1.657961904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:44.654291336 +0000 UTC m=+1651.146107882" watchObservedRunningTime="2026-10-30 21:04:44.657961904 +0000 UTC m=+1651.149778449"
Oct 30 21:04:45.157943 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:45.157911    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:54.158632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:54.158587    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:54.171520 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:54.171483    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:54.768426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:04:54.768339    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:04:55.158501 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:04:55.158458    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:05:09.032309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:09.032262    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"]
Oct 30 21:05:09.039168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:09.039132    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad8d75b-c2j69"]
Oct 30 21:05:09.663425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:09.663393    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ba0110-f784-4317-abe4-fb0d7545bf25" path="/var/lib/kubelet/pods/f8ba0110-f784-4317-abe4-fb0d7545bf25/volumes"
Oct 30 21:05:15.552018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:15.551979    4810 scope.go:122] "RemoveContainer" containerID="053ee27fa741e9c66f96cd9b6e591adcd4c898f6b5eceb84295e8147c2245252"
Oct 30 21:05:15.562778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:15.562760    4810 scope.go:122] "RemoveContainer" containerID="30cecadba1bbbf6efb74f3823d585a5a665327304485f8b8c1b4437d1b4d92a8"
Oct 30 21:05:15.574155 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:15.574135    4810 scope.go:122] "RemoveContainer" containerID="3939c71c94337b74f178de18b330b26464c8fb1a082e3d76992199e6992ba88a"
Oct 30 21:05:16.760717 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:16.760683    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:05:21.028525 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:21.028493    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"]
Oct 30 21:05:21.035343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:21.035309    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e40f1db9-k28bz"]
Oct 30 21:05:21.663792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:21.663759    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11777510-934a-4d08-96bf-3e00a8ae19c7" path="/var/lib/kubelet/pods/11777510-934a-4d08-96bf-3e00a8ae19c7/volumes"
Oct 30 21:05:23.758104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:23.758065    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"]
Oct 30 21:05:23.758615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:23.758342    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy" containerID="cri-o://15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6" gracePeriod=30
Oct 30 21:05:25.169976 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:05:25.169940    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:05:29.055222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.055195    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 21:05:29.092265 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.092232    4810 generic.go:356] "Generic (PLEG): container finished" podID="16b83400-6692-4ba7-b59f-14c215517039" containerID="15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6" exitCode=0
Oct 30 21:05:29.092421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.092377    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" event={"ID":"16b83400-6692-4ba7-b59f-14c215517039","Type":"ContainerDied","Data":"15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6"}
Oct 30 21:05:29.092510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.092491    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l" event={"ID":"16b83400-6692-4ba7-b59f-14c215517039","Type":"ContainerDied","Data":"a837e0e80aaa8260a09c92006e499245e1e277c04b3c45d249db4cc502042df8"}
Oct 30 21:05:29.092573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.092528    4810 scope.go:122] "RemoveContainer" containerID="15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6"
Oct 30 21:05:29.092707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.092692    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"
Oct 30 21:05:29.124092 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.124064    4810 scope.go:122] "RemoveContainer" containerID="15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6"
Oct 30 21:05:29.124436 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:05:29.124409    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6\": container with ID starting with 15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6 not found: ID does not exist" containerID="15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6"
Oct 30 21:05:29.124526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.124447    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6"} err="failed to get container status \"15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6\": rpc error: code = NotFound desc = could not find container \"15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6\": container with ID starting with 15aa64124e12b202220e5956614bf3e7f3b4c1b9ad29d1875fed64117864b0e6 not found: ID does not exist"
Oct 30 21:05:29.195502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.195462    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-socket\") pod \"16b83400-6692-4ba7-b59f-14c215517039\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") "
Oct 30 21:05:29.195691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.195540    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-envoy\") pod \"16b83400-6692-4ba7-b59f-14c215517039\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") "
Oct 30 21:05:29.195691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.195583    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-credential-socket\") pod \"16b83400-6692-4ba7-b59f-14c215517039\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") "
Oct 30 21:05:29.195691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.195647    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-kube-api-access-99plp\" (UniqueName: \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-kube-api-access-99plp\") pod \"16b83400-6692-4ba7-b59f-14c215517039\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") "
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.195712    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-istio-token\" (UniqueName: \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-istio-token\") pod \"16b83400-6692-4ba7-b59f-14c215517039\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") "
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.195803    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16b83400-6692-4ba7-b59f-14c215517039-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b83400-6692-4ba7-b59f-14c215517039-istio-podinfo\") pod \"16b83400-6692-4ba7-b59f-14c215517039\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") "
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.195840    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-data\") pod \"16b83400-6692-4ba7-b59f-14c215517039\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") "
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.195995    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b83400-6692-4ba7-b59f-14c215517039-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b83400-6692-4ba7-b59f-14c215517039-istiod-ca-cert\") pod \"16b83400-6692-4ba7-b59f-14c215517039\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") "
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.196044    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-certs\") pod \"16b83400-6692-4ba7-b59f-14c215517039\" (UID: \"16b83400-6692-4ba7-b59f-14c215517039\") "
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.196372    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-credential-socket" pod "16b83400-6692-4ba7-b59f-14c215517039" (UID: "16b83400-6692-4ba7-b59f-14c215517039"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.196460    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-data" pod "16b83400-6692-4ba7-b59f-14c215517039" (UID: "16b83400-6692-4ba7-b59f-14c215517039"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.196590    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-certs" pod "16b83400-6692-4ba7-b59f-14c215517039" (UID: "16b83400-6692-4ba7-b59f-14c215517039"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.197019    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b83400-6692-4ba7-b59f-14c215517039-istiod-ca-cert" pod "16b83400-6692-4ba7-b59f-14c215517039" (UID: "16b83400-6692-4ba7-b59f-14c215517039"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.197076    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.197101    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:29.197418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.197152    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:29.198664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.198195    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-socket" pod "16b83400-6692-4ba7-b59f-14c215517039" (UID: "16b83400-6692-4ba7-b59f-14c215517039"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:29.199262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.199232    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16b83400-6692-4ba7-b59f-14c215517039-istio-podinfo" pod "16b83400-6692-4ba7-b59f-14c215517039" (UID: "16b83400-6692-4ba7-b59f-14c215517039"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:05:29.200306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.200266    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-envoy" pod "16b83400-6692-4ba7-b59f-14c215517039" (UID: "16b83400-6692-4ba7-b59f-14c215517039"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:29.202265 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.202241    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-istio-token" pod "16b83400-6692-4ba7-b59f-14c215517039" (UID: "16b83400-6692-4ba7-b59f-14c215517039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:29.202345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.202289    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-kube-api-access-99plp" pod "16b83400-6692-4ba7-b59f-14c215517039" (UID: "16b83400-6692-4ba7-b59f-14c215517039"). InnerVolumeSpecName "kube-api-access-99plp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:29.298600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.298543    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b83400-6692-4ba7-b59f-14c215517039-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:29.298600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.298567    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b83400-6692-4ba7-b59f-14c215517039-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:29.298600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.298577    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:29.298600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.298586    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b83400-6692-4ba7-b59f-14c215517039-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:29.298600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.298596    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99plp\" (UniqueName: \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-kube-api-access-99plp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:29.298600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.298605    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b83400-6692-4ba7-b59f-14c215517039-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:29.422646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.422615    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"]
Oct 30 21:05:29.429034 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.429011    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cwl6l"]
Oct 30 21:05:29.665198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:29.665164    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b83400-6692-4ba7-b59f-14c215517039" path="/var/lib/kubelet/pods/16b83400-6692-4ba7-b59f-14c215517039/volumes"
Oct 30 21:05:31.055079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:31.055045    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"]
Oct 30 21:05:31.063816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:31.063583    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e35d74-vk7nd"]
Oct 30 21:05:31.665212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:31.665184    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc30df6-d7dc-437e-9bba-a1053463abea" path="/var/lib/kubelet/pods/ffc30df6-d7dc-437e-9bba-a1053463abea/volumes"
Oct 30 21:05:31.988018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:31.987908    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"]
Oct 30 21:05:31.988820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:31.988797    4810 cpu_manager.go:414] "containerMap: removing container" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy"
Oct 30 21:05:31.988820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:31.988816    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy"
Oct 30 21:05:31.989003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:31.988974    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b83400-6692-4ba7-b59f-14c215517039" containerName="istio-proxy"
Oct 30 21:05:31.994760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:31.994734    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.001894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.001866    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"]
Oct 30 21:05:32.004078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.004018    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-89d993\"/\"kube-root-ca.crt\""
Oct 30 21:05:32.004078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.004031    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-89d993\"/\"openshift-service-ca.crt\""
Oct 30 21:05:32.004339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.004111    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-89d993\"/\"default-dockercfg-dzj8g\""
Oct 30 21:05:32.004339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.004158    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-89d993\"/\"praxis-route-89d993-kserve-self-signed-certs\""
Oct 30 21:05:32.053671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.053644    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tmp-dir\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.053829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.053718    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tls-certs\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.053902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.053827    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-model-cache\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.053902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.053851    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-home\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.054084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.054063    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-dshm\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.054146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.054115    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmljl\" (UniqueName: \"kubernetes.io/projected/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-kube-api-access-mmljl\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.157391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.157355    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tls-certs\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.158470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.158442    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-model-cache\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.158618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.158483    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-home\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.158685 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.158625    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-dshm\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.158685 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.158667    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmljl\" (UniqueName: \"kubernetes.io/projected/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-kube-api-access-mmljl\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.158836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.158817    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tmp-dir\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.159327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.159303    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-home\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.159551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.159312    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tmp-dir\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.159954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.159934    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-model-cache\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.199883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.196963    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmljl\" (UniqueName: \"kubernetes.io/projected/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-kube-api-access-mmljl\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.228791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.228756    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tls-certs\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.237832 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.230171    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-dshm\") pod \"praxis-route-89d993-kserve-9b456cf57-hrrxl\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") " pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.317917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.317825    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:32.618395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:32.618364    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"]
Oct 30 21:05:32.619902 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:05:32.619874    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac0d2289_daf5_4fc4_92bc_ff73548a2d73.slice/crio-6904fccd609645d034aaa3308f39b15d9473918e0bbbd6d0d5fcdf4819065521 WatchSource:0}: Error finding container 6904fccd609645d034aaa3308f39b15d9473918e0bbbd6d0d5fcdf4819065521: Status 404 returned error can't find the container with id 6904fccd609645d034aaa3308f39b15d9473918e0bbbd6d0d5fcdf4819065521
Oct 30 21:05:33.136433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:33.136394    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl" event={"ID":"ac0d2289-daf5-4fc4-92bc-ff73548a2d73","Type":"ContainerStarted","Data":"bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc"}
Oct 30 21:05:33.136640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:33.136446    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl" event={"ID":"ac0d2289-daf5-4fc4-92bc-ff73548a2d73","Type":"ContainerStarted","Data":"6904fccd609645d034aaa3308f39b15d9473918e0bbbd6d0d5fcdf4819065521"}
Oct 30 21:05:33.156886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:33.156823    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl" podStartSLOduration=2.156809495 podStartE2EDuration="2.156809495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:33.153157053 +0000 UTC m=+1699.644973614" watchObservedRunningTime="2026-10-30 21:05:33.156809495 +0000 UTC m=+1699.648626039"
Oct 30 21:05:33.318836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:33.318805    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:35.025572 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:35.025532    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"]
Oct 30 21:05:35.030032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:35.030005    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c31b208-fn2ct"]
Oct 30 21:05:35.664064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:35.664032    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66509f8e-0592-44a8-b268-d33317745944" path="/var/lib/kubelet/pods/66509f8e-0592-44a8-b268-d33317745944/volumes"
Oct 30 21:05:36.026350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:36.026272    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"]
Oct 30 21:05:36.030296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:36.030268    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8871d183-kwhdr"]
Oct 30 21:05:37.664728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:37.664696    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b" path="/var/lib/kubelet/pods/13b54b68-d9a3-4fa4-ba54-ac6a47eb1d8b/volumes"
Oct 30 21:05:42.318731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:42.318692    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:42.332161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:42.332137    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:43.245359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:43.245332    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:05:44.033027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:44.032988    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"]
Oct 30 21:05:44.038945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:44.038915    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6bfc82-7wmpw"]
Oct 30 21:05:45.663967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:45.663937    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250cf721-68c9-42de-b624-7d1a930b4867" path="/var/lib/kubelet/pods/250cf721-68c9-42de-b624-7d1a930b4867/volumes"
Oct 30 21:05:48.028111 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:48.028061    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"]
Oct 30 21:05:48.032835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:48.032805    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d1af73b-2jdb2"]
Oct 30 21:05:49.115744 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.115695    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"]
Oct 30 21:05:49.122330 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.122301    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91a3ee10-kbkj7"]
Oct 30 21:05:49.128369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.128326    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"]
Oct 30 21:05:49.135227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.135203    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.139467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.139442    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:05:49.140305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.140280    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"]
Oct 30 21:05:49.140519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.140344    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:05:49.243674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.243640    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p59vv\" (UniqueName: \"kubernetes.io/projected/67536d32-e34c-4b6e-807f-8b1b630ce07f-kube-api-access-p59vv\") pod \"maas-api-revoke-sub-140c620d-kzx8x\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.243868 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.243832    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140c620d-kzx8x\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.243984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.243964    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140c620d-kzx8x\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.345720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.345688    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p59vv\" (UniqueName: \"kubernetes.io/projected/67536d32-e34c-4b6e-807f-8b1b630ce07f-kube-api-access-p59vv\") pod \"maas-api-revoke-sub-140c620d-kzx8x\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.345927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.345792    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140c620d-kzx8x\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.345927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.345905    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140c620d-kzx8x\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.346591 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.346564    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140c620d-kzx8x\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.348254 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.348236    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140c620d-kzx8x\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.353978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.353952    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p59vv\" (UniqueName: \"kubernetes.io/projected/67536d32-e34c-4b6e-807f-8b1b630ce07f-kube-api-access-p59vv\") pod \"maas-api-revoke-sub-140c620d-kzx8x\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.461175 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.461141    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:49.669624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.669589    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6" path="/var/lib/kubelet/pods/643f6fef-b4e5-420e-b7ed-93a4b7a8f3d6/volumes"
Oct 30 21:05:49.670206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.670184    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d225e5d-b49e-4b42-b81d-e1bf519349af" path="/var/lib/kubelet/pods/7d225e5d-b49e-4b42-b81d-e1bf519349af/volumes"
Oct 30 21:05:49.753471 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:05:49.753434    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67536d32_e34c_4b6e_807f_8b1b630ce07f.slice/crio-ceba19092d4292a063112e1d1ff19008efc8ba769eeb590d0c95d40d15296365 WatchSource:0}: Error finding container ceba19092d4292a063112e1d1ff19008efc8ba769eeb590d0c95d40d15296365: Status 404 returned error can't find the container with id ceba19092d4292a063112e1d1ff19008efc8ba769eeb590d0c95d40d15296365
Oct 30 21:05:49.753628 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:49.753494    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"]
Oct 30 21:05:50.319909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:50.316611    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x" event={"ID":"67536d32-e34c-4b6e-807f-8b1b630ce07f","Type":"ContainerStarted","Data":"30a4dda8f7198b83c7af25b23bcdd5955dfe68638a894b91db221304e4e8661d"}
Oct 30 21:05:50.319909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:50.316682    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x" event={"ID":"67536d32-e34c-4b6e-807f-8b1b630ce07f","Type":"ContainerStarted","Data":"ceba19092d4292a063112e1d1ff19008efc8ba769eeb590d0c95d40d15296365"}
Oct 30 21:05:50.338845 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:50.338725    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x" podStartSLOduration=1.338707248 podStartE2EDuration="1.338707248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:50.333984331 +0000 UTC m=+1716.825800871" watchObservedRunningTime="2026-10-30 21:05:50.338707248 +0000 UTC m=+1716.830523794"
Oct 30 21:05:51.327814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:51.327774    4810 generic.go:356] "Generic (PLEG): container finished" podID="67536d32-e34c-4b6e-807f-8b1b630ce07f" containerID="30a4dda8f7198b83c7af25b23bcdd5955dfe68638a894b91db221304e4e8661d" exitCode=0
Oct 30 21:05:51.328295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:51.327866    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x" event={"ID":"67536d32-e34c-4b6e-807f-8b1b630ce07f","Type":"ContainerDied","Data":"30a4dda8f7198b83c7af25b23bcdd5955dfe68638a894b91db221304e4e8661d"}
Oct 30 21:05:52.541615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:52.541583    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:52.605701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:52.605613    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-openshift-ca\") pod \"67536d32-e34c-4b6e-807f-8b1b630ce07f\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") "
Oct 30 21:05:52.605897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:52.605799    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67536d32-e34c-4b6e-807f-8b1b630ce07f-kube-api-access-p59vv\" (UniqueName: \"kubernetes.io/projected/67536d32-e34c-4b6e-807f-8b1b630ce07f-kube-api-access-p59vv\") pod \"67536d32-e34c-4b6e-807f-8b1b630ce07f\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") "
Oct 30 21:05:52.605897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:52.605889    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-xks-ca\") pod \"67536d32-e34c-4b6e-807f-8b1b630ce07f\" (UID: \"67536d32-e34c-4b6e-807f-8b1b630ce07f\") "
Oct 30 21:05:52.606052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:52.606030    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-openshift-ca" pod "67536d32-e34c-4b6e-807f-8b1b630ce07f" (UID: "67536d32-e34c-4b6e-807f-8b1b630ce07f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:52.606515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:52.606490    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:52.609264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:52.609224    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-xks-ca" pod "67536d32-e34c-4b6e-807f-8b1b630ce07f" (UID: "67536d32-e34c-4b6e-807f-8b1b630ce07f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:05:52.609526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:52.609486    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67536d32-e34c-4b6e-807f-8b1b630ce07f-kube-api-access-p59vv" pod "67536d32-e34c-4b6e-807f-8b1b630ce07f" (UID: "67536d32-e34c-4b6e-807f-8b1b630ce07f"). InnerVolumeSpecName "kube-api-access-p59vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:52.708134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:52.708102    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67536d32-e34c-4b6e-807f-8b1b630ce07f-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:52.708134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:52.708130    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p59vv\" (UniqueName: \"kubernetes.io/projected/67536d32-e34c-4b6e-807f-8b1b630ce07f-kube-api-access-p59vv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:05:53.346661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:53.346628    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x" event={"ID":"67536d32-e34c-4b6e-807f-8b1b630ce07f","Type":"ContainerDied","Data":"ceba19092d4292a063112e1d1ff19008efc8ba769eeb590d0c95d40d15296365"}
Oct 30 21:05:53.346849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:53.346677    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceba19092d4292a063112e1d1ff19008efc8ba769eeb590d0c95d40d15296365"
Oct 30 21:05:53.346849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:53.346641    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"
Oct 30 21:05:55.160194 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:05:55.160160    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:05:59.335036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.335001    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"]
Oct 30 21:05:59.335661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.335644    4810 cpu_manager.go:414] "containerMap: removing container" podUID="67536d32-e34c-4b6e-807f-8b1b630ce07f" containerName="revoke-keys"
Oct 30 21:05:59.335707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.335661    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67536d32-e34c-4b6e-807f-8b1b630ce07f" containerName="revoke-keys"
Oct 30 21:05:59.335818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.335808    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="67536d32-e34c-4b6e-807f-8b1b630ce07f" containerName="revoke-keys"
Oct 30 21:05:59.339853 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.339831    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.344138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.344113    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:05:59.344256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.344198    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:05:59.348074 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.348055    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"]
Oct 30 21:05:59.524490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.524455    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba83cc37-jzv6k\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.524668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.524506    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba83cc37-jzv6k\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.524668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.524614    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7r7\" (UniqueName: \"kubernetes.io/projected/cca399a2-5ae1-4dd6-836d-bf84cb03549e-kube-api-access-kr7r7\") pod \"maas-api-revoke-sub-ba83cc37-jzv6k\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.626179 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.626085    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba83cc37-jzv6k\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.626179 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.626160    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba83cc37-jzv6k\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.626445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.626236    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7r7\" (UniqueName: \"kubernetes.io/projected/cca399a2-5ae1-4dd6-836d-bf84cb03549e-kube-api-access-kr7r7\") pod \"maas-api-revoke-sub-ba83cc37-jzv6k\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.626903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.626877    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba83cc37-jzv6k\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.635594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.635562    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba83cc37-jzv6k\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.637186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.637160    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7r7\" (UniqueName: \"kubernetes.io/projected/cca399a2-5ae1-4dd6-836d-bf84cb03549e-kube-api-access-kr7r7\") pod \"maas-api-revoke-sub-ba83cc37-jzv6k\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.656651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.656622    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:05:59.940896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:05:59.940844    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"]
Oct 30 21:05:59.941103 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:05:59.941083    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca399a2_5ae1_4dd6_836d_bf84cb03549e.slice/crio-c2129054ef345a46daa3abf7c6562d1eddefa064d32154c1f1245962034b09fc WatchSource:0}: Error finding container c2129054ef345a46daa3abf7c6562d1eddefa064d32154c1f1245962034b09fc: Status 404 returned error can't find the container with id c2129054ef345a46daa3abf7c6562d1eddefa064d32154c1f1245962034b09fc
Oct 30 21:06:00.034641 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:00.034614    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"]
Oct 30 21:06:00.038355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:00.038332    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cca24a1-ctsc2"]
Oct 30 21:06:00.409360 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:00.409330    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k" event={"ID":"cca399a2-5ae1-4dd6-836d-bf84cb03549e","Type":"ContainerStarted","Data":"c725f9668c3060066f4ee13d65a5b85cfecbbd4fd67bad3beb88fd202fd47179"}
Oct 30 21:06:00.409681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:00.409422    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k" event={"ID":"cca399a2-5ae1-4dd6-836d-bf84cb03549e","Type":"ContainerStarted","Data":"c2129054ef345a46daa3abf7c6562d1eddefa064d32154c1f1245962034b09fc"}
Oct 30 21:06:00.436355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:00.436310    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k" podStartSLOduration=1.436296467 podStartE2EDuration="1.436296467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:00.433388299 +0000 UTC m=+1726.925204841" watchObservedRunningTime="2026-10-30 21:06:00.436296467 +0000 UTC m=+1726.928113011"
Oct 30 21:06:01.419097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:01.419054    4810 generic.go:356] "Generic (PLEG): container finished" podID="cca399a2-5ae1-4dd6-836d-bf84cb03549e" containerID="c725f9668c3060066f4ee13d65a5b85cfecbbd4fd67bad3beb88fd202fd47179" exitCode=0
Oct 30 21:06:01.419596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:01.419134    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k" event={"ID":"cca399a2-5ae1-4dd6-836d-bf84cb03549e","Type":"ContainerDied","Data":"c725f9668c3060066f4ee13d65a5b85cfecbbd4fd67bad3beb88fd202fd47179"}
Oct 30 21:06:01.663494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:01.663461    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce3e833-d075-43d6-b477-774d5f680a1f" path="/var/lib/kubelet/pods/5ce3e833-d075-43d6-b477-774d5f680a1f/volumes"
Oct 30 21:06:02.631285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:02.631251    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:06:02.666160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:02.666124    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca399a2-5ae1-4dd6-836d-bf84cb03549e-kube-api-access-kr7r7\" (UniqueName: \"kubernetes.io/projected/cca399a2-5ae1-4dd6-836d-bf84cb03549e-kube-api-access-kr7r7\") pod \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") "
Oct 30 21:06:02.666356 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:02.666325    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-openshift-ca\") pod \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") "
Oct 30 21:06:02.666431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:02.666371    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-xks-ca\") pod \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\" (UID: \"cca399a2-5ae1-4dd6-836d-bf84cb03549e\") "
Oct 30 21:06:02.667471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:02.667439    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-openshift-ca" pod "cca399a2-5ae1-4dd6-836d-bf84cb03549e" (UID: "cca399a2-5ae1-4dd6-836d-bf84cb03549e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:02.669087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:02.669058    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca399a2-5ae1-4dd6-836d-bf84cb03549e-kube-api-access-kr7r7" pod "cca399a2-5ae1-4dd6-836d-bf84cb03549e" (UID: "cca399a2-5ae1-4dd6-836d-bf84cb03549e"). InnerVolumeSpecName "kube-api-access-kr7r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:02.669826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:02.669798    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-xks-ca" pod "cca399a2-5ae1-4dd6-836d-bf84cb03549e" (UID: "cca399a2-5ae1-4dd6-836d-bf84cb03549e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:02.769181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:02.769144    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:02.769181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:02.769171    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca399a2-5ae1-4dd6-836d-bf84cb03549e-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:02.769181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:02.769183    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr7r7\" (UniqueName: \"kubernetes.io/projected/cca399a2-5ae1-4dd6-836d-bf84cb03549e-kube-api-access-kr7r7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.029128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:03.029050    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"]
Oct 30 21:06:03.035437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:03.035408    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df431714-sl8kg"]
Oct 30 21:06:03.439415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:03.439387    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"
Oct 30 21:06:03.439614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:03.439416    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k" event={"ID":"cca399a2-5ae1-4dd6-836d-bf84cb03549e","Type":"ContainerDied","Data":"c2129054ef345a46daa3abf7c6562d1eddefa064d32154c1f1245962034b09fc"}
Oct 30 21:06:03.439614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:03.439462    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2129054ef345a46daa3abf7c6562d1eddefa064d32154c1f1245962034b09fc"
Oct 30 21:06:03.665096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:03.665062    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5871f831-513b-4c57-9be3-2b5b296dbe00" path="/var/lib/kubelet/pods/5871f831-513b-4c57-9be3-2b5b296dbe00/volumes"
Oct 30 21:06:11.155183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:11.155143    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"]
Oct 30 21:06:15.156721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:15.156692    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:06:15.624848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:15.624773    4810 scope.go:122] "RemoveContainer" containerID="652d846199aed87fccabece38986e166f7ed1d525931851692c2ba60de50cb4f"
Oct 30 21:06:15.638882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:15.638847    4810 scope.go:122] "RemoveContainer" containerID="bef0e1dd16d15adfc90cd4f0d5ab1d622342fd0c7c0b38647a6e7f4b2e0ce838"
Oct 30 21:06:15.648704 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:15.648683    4810 scope.go:122] "RemoveContainer" containerID="c23094d4f08cb58052d945e28a0dcaaa0ebfbfff665b970048a1932190d64226"
Oct 30 21:06:15.658642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:15.658624    4810 scope.go:122] "RemoveContainer" containerID="1a66f6e042e5ad12f3e1658a5f7688e508a7160239764fa6c07bfaef5e1e5b25"
Oct 30 21:06:16.031234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:16.031192    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"]
Oct 30 21:06:16.035985 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:16.035952    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"]
Oct 30 21:06:16.039444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:16.039418    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d8571b-zjg5w"]
Oct 30 21:06:16.044981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:16.044955    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5550ae3-4qptm"]
Oct 30 21:06:17.665221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:17.665186    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4aa1d7-fb3f-4739-b215-ef573d30d152" path="/var/lib/kubelet/pods/4a4aa1d7-fb3f-4739-b215-ef573d30d152/volumes"
Oct 30 21:06:17.665935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:17.665899    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a971368c-d08f-42c3-a2e9-8ece37f71254" path="/var/lib/kubelet/pods/a971368c-d08f-42c3-a2e9-8ece37f71254/volumes"
Oct 30 21:06:20.702163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:20.702119    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"]
Oct 30 21:06:20.946806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:20.946768    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-b45643"]
Oct 30 21:06:20.947532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:20.947517    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cca399a2-5ae1-4dd6-836d-bf84cb03549e" containerName="revoke-keys"
Oct 30 21:06:20.947585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:20.947534    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca399a2-5ae1-4dd6-836d-bf84cb03549e" containerName="revoke-keys"
Oct 30 21:06:20.947671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:20.947661    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca399a2-5ae1-4dd6-836d-bf84cb03549e" containerName="revoke-keys"
Oct 30 21:06:20.953530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:20.953460    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-b45643"
Oct 30 21:06:20.960589 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:20.960554    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-b45643"]
Oct 30 21:06:20.961607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:20.961583    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrzx8\""
Oct 30 21:06:21.051831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:21.051798    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vftrq\" (UniqueName: \"kubernetes.io/projected/cb4d4a26-009c-40ea-88ea-a3f43a678a0a-kube-api-access-vftrq\") pod \"test-curl-7609-b45643\" (UID: \"cb4d4a26-009c-40ea-88ea-a3f43a678a0a\") " pod="opendatahub/test-curl-7609-b45643"
Oct 30 21:06:21.153621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:21.153579    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vftrq\" (UniqueName: \"kubernetes.io/projected/cb4d4a26-009c-40ea-88ea-a3f43a678a0a-kube-api-access-vftrq\") pod \"test-curl-7609-b45643\" (UID: \"cb4d4a26-009c-40ea-88ea-a3f43a678a0a\") " pod="opendatahub/test-curl-7609-b45643"
Oct 30 21:06:21.164551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:21.164518    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vftrq\" (UniqueName: \"kubernetes.io/projected/cb4d4a26-009c-40ea-88ea-a3f43a678a0a-kube-api-access-vftrq\") pod \"test-curl-7609-b45643\" (UID: \"cb4d4a26-009c-40ea-88ea-a3f43a678a0a\") " pod="opendatahub/test-curl-7609-b45643"
Oct 30 21:06:21.289676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:21.289603    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-b45643"
Oct 30 21:06:21.603406 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:06:21.603373    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb4d4a26_009c_40ea_88ea_a3f43a678a0a.slice/crio-d0757901a7d11d899e784350740f22d30f7ff98a2f4a4642e23980368ce75e53 WatchSource:0}: Error finding container d0757901a7d11d899e784350740f22d30f7ff98a2f4a4642e23980368ce75e53: Status 404 returned error can't find the container with id d0757901a7d11d899e784350740f22d30f7ff98a2f4a4642e23980368ce75e53
Oct 30 21:06:21.604715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:21.604691    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-b45643"]
Oct 30 21:06:21.634873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:21.632774    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-b45643" event={"ID":"cb4d4a26-009c-40ea-88ea-a3f43a678a0a","Type":"ContainerStarted","Data":"d0757901a7d11d899e784350740f22d30f7ff98a2f4a4642e23980368ce75e53"}
Oct 30 21:06:22.645355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:22.645321    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-b45643" event={"ID":"cb4d4a26-009c-40ea-88ea-a3f43a678a0a","Type":"ContainerStarted","Data":"64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609"}
Oct 30 21:06:22.661785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:22.661739    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7609-b45643" podStartSLOduration=2.661723459 podStartE2EDuration="2.661723459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:22.660738636 +0000 UTC m=+1749.152555182" watchObservedRunningTime="2026-10-30 21:06:22.661723459 +0000 UTC m=+1749.153540008"
Oct 30 21:06:23.324428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.324396    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-b45643"]
Oct 30 21:06:23.329182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.329153    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-b45643"]
Oct 30 21:06:23.485419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.485384    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-8b05dd"]
Oct 30 21:06:23.486421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.486403    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cb4d4a26-009c-40ea-88ea-a3f43a678a0a" containerName="test-curl-7609-b45643"
Oct 30 21:06:23.486480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.486424    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb4d4a26-009c-40ea-88ea-a3f43a678a0a" containerName="test-curl-7609-b45643"
Oct 30 21:06:23.486594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.486583    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb4d4a26-009c-40ea-88ea-a3f43a678a0a" containerName="test-curl-7609-b45643"
Oct 30 21:06:23.493514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.492700    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-8b05dd"
Oct 30 21:06:23.495168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.495142    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-8b05dd"]
Oct 30 21:06:23.585623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.585554    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzt2x\" (UniqueName: \"kubernetes.io/projected/2179cd4c-1e2d-4b1f-8a10-fa1fda488373-kube-api-access-lzt2x\") pod \"test-curl-7609-8b05dd\" (UID: \"2179cd4c-1e2d-4b1f-8a10-fa1fda488373\") " pod="opendatahub/test-curl-7609-8b05dd"
Oct 30 21:06:23.686644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.686617    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzt2x\" (UniqueName: \"kubernetes.io/projected/2179cd4c-1e2d-4b1f-8a10-fa1fda488373-kube-api-access-lzt2x\") pod \"test-curl-7609-8b05dd\" (UID: \"2179cd4c-1e2d-4b1f-8a10-fa1fda488373\") " pod="opendatahub/test-curl-7609-8b05dd"
Oct 30 21:06:23.696634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.696610    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzt2x\" (UniqueName: \"kubernetes.io/projected/2179cd4c-1e2d-4b1f-8a10-fa1fda488373-kube-api-access-lzt2x\") pod \"test-curl-7609-8b05dd\" (UID: \"2179cd4c-1e2d-4b1f-8a10-fa1fda488373\") " pod="opendatahub/test-curl-7609-8b05dd"
Oct 30 21:06:23.809444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:23.809411    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-8b05dd"
Oct 30 21:06:24.096843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:24.096812    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-8b05dd"]
Oct 30 21:06:24.098190 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:06:24.098162    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2179cd4c_1e2d_4b1f_8a10_fa1fda488373.slice/crio-6ad7736bc0cbf7e189c69f33f04353c3e14436e6f4ed97c5ceb35f985d240f5f WatchSource:0}: Error finding container 6ad7736bc0cbf7e189c69f33f04353c3e14436e6f4ed97c5ceb35f985d240f5f: Status 404 returned error can't find the container with id 6ad7736bc0cbf7e189c69f33f04353c3e14436e6f4ed97c5ceb35f985d240f5f
Oct 30 21:06:24.663170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:24.663132    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-8b05dd" event={"ID":"2179cd4c-1e2d-4b1f-8a10-fa1fda488373","Type":"ContainerStarted","Data":"72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2"}
Oct 30 21:06:24.663813 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:24.663178    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-8b05dd" event={"ID":"2179cd4c-1e2d-4b1f-8a10-fa1fda488373","Type":"ContainerStarted","Data":"6ad7736bc0cbf7e189c69f33f04353c3e14436e6f4ed97c5ceb35f985d240f5f"}
Oct 30 21:06:24.663813 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:24.663280    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7609-b45643" podUID="cb4d4a26-009c-40ea-88ea-a3f43a678a0a" containerName="test-curl-7609-b45643" containerID="cri-o://64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609" gracePeriod=2
Oct 30 21:06:24.681951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:24.681835    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7609-8b05dd" podStartSLOduration=1.681816311 podStartE2EDuration="1.681816311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:24.676280691 +0000 UTC m=+1751.168097237" watchObservedRunningTime="2026-10-30 21:06:24.681816311 +0000 UTC m=+1751.173632855"
Oct 30 21:06:25.165089 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:06:25.165050    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:06:25.380590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.380397    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-8b05dd"]
Oct 30 21:06:25.402506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.388635    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-8b05dd"]
Oct 30 21:06:25.618195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.618105    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-703e17"]
Oct 30 21:06:25.619335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.619308    4810 cpu_manager.go:414] "containerMap: removing container" podUID="2179cd4c-1e2d-4b1f-8a10-fa1fda488373" containerName="test-curl-7609-8b05dd"
Oct 30 21:06:25.619335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.619333    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2179cd4c-1e2d-4b1f-8a10-fa1fda488373" containerName="test-curl-7609-8b05dd"
Oct 30 21:06:25.619995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.619969    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="2179cd4c-1e2d-4b1f-8a10-fa1fda488373" containerName="test-curl-7609-8b05dd"
Oct 30 21:06:25.624407 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.624387    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-703e17"]
Oct 30 21:06:25.624497 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.624483    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-703e17"
Oct 30 21:06:25.731809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.731773    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8dv7\" (UniqueName: \"kubernetes.io/projected/8a1bbe07-995e-44ab-8a35-24f0137e0f97-kube-api-access-h8dv7\") pod \"test-curl-7609-703e17\" (UID: \"8a1bbe07-995e-44ab-8a35-24f0137e0f97\") " pod="opendatahub/test-curl-7609-703e17"
Oct 30 21:06:25.832966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.832935    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8dv7\" (UniqueName: \"kubernetes.io/projected/8a1bbe07-995e-44ab-8a35-24f0137e0f97-kube-api-access-h8dv7\") pod \"test-curl-7609-703e17\" (UID: \"8a1bbe07-995e-44ab-8a35-24f0137e0f97\") " pod="opendatahub/test-curl-7609-703e17"
Oct 30 21:06:25.842252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.842222    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8dv7\" (UniqueName: \"kubernetes.io/projected/8a1bbe07-995e-44ab-8a35-24f0137e0f97-kube-api-access-h8dv7\") pod \"test-curl-7609-703e17\" (UID: \"8a1bbe07-995e-44ab-8a35-24f0137e0f97\") " pod="opendatahub/test-curl-7609-703e17"
Oct 30 21:06:25.942171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:25.942145    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-703e17"
Oct 30 21:06:26.193775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.192144    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v" podUID="3002922a-ae83-42b9-8d51-08d606c5433f" containerName="main" containerID="cri-o://c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8" gracePeriod=45
Oct 30 21:06:26.243931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.243899    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-703e17"]
Oct 30 21:06:26.281762 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.281686    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v" podUID="3002922a-ae83-42b9-8d51-08d606c5433f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.193:8000/ready\": dial tcp 10.132.0.193:8000: connect: connection refused"
Oct 30 21:06:26.467971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.467947    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:06:26.543119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.543084    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-model-cache\") pod \"3002922a-ae83-42b9-8d51-08d606c5433f\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") "
Oct 30 21:06:26.543309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.543283    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-tmp-dir\") pod \"3002922a-ae83-42b9-8d51-08d606c5433f\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") "
Oct 30 21:06:26.543383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.543353    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-model-cache" pod "3002922a-ae83-42b9-8d51-08d606c5433f" (UID: "3002922a-ae83-42b9-8d51-08d606c5433f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:26.543491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.543377    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3002922a-ae83-42b9-8d51-08d606c5433f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3002922a-ae83-42b9-8d51-08d606c5433f-tls-certs\") pod \"3002922a-ae83-42b9-8d51-08d606c5433f\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") "
Oct 30 21:06:26.543491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.543480    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-dshm\") pod \"3002922a-ae83-42b9-8d51-08d606c5433f\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") "
Oct 30 21:06:26.543614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.543540    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-tmp-dir" pod "3002922a-ae83-42b9-8d51-08d606c5433f" (UID: "3002922a-ae83-42b9-8d51-08d606c5433f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:26.543614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.543607    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3002922a-ae83-42b9-8d51-08d606c5433f-kube-api-access-zwlwc\" (UniqueName: \"kubernetes.io/projected/3002922a-ae83-42b9-8d51-08d606c5433f-kube-api-access-zwlwc\") pod \"3002922a-ae83-42b9-8d51-08d606c5433f\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") "
Oct 30 21:06:26.543732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.543693    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-home\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-home\") pod \"3002922a-ae83-42b9-8d51-08d606c5433f\" (UID: \"3002922a-ae83-42b9-8d51-08d606c5433f\") "
Oct 30 21:06:26.544414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.544100    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-home" pod "3002922a-ae83-42b9-8d51-08d606c5433f" (UID: "3002922a-ae83-42b9-8d51-08d606c5433f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:26.544414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.544355    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:26.544414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.544374    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:26.544414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.544389    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:26.545946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.545918    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3002922a-ae83-42b9-8d51-08d606c5433f-tls-certs" pod "3002922a-ae83-42b9-8d51-08d606c5433f" (UID: "3002922a-ae83-42b9-8d51-08d606c5433f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:26.546235 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.546195    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3002922a-ae83-42b9-8d51-08d606c5433f-kube-api-access-zwlwc" pod "3002922a-ae83-42b9-8d51-08d606c5433f" (UID: "3002922a-ae83-42b9-8d51-08d606c5433f"). InnerVolumeSpecName "kube-api-access-zwlwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:26.546352 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.546335    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-dshm" pod "3002922a-ae83-42b9-8d51-08d606c5433f" (UID: "3002922a-ae83-42b9-8d51-08d606c5433f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:26.646181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.646146    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwlwc\" (UniqueName: \"kubernetes.io/projected/3002922a-ae83-42b9-8d51-08d606c5433f-kube-api-access-zwlwc\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:26.646181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.646180    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3002922a-ae83-42b9-8d51-08d606c5433f-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:26.646353 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.646194    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3002922a-ae83-42b9-8d51-08d606c5433f-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:26.680205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.680178    4810 generic.go:356] "Generic (PLEG): container finished" podID="3002922a-ae83-42b9-8d51-08d606c5433f" containerID="c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8" exitCode=0
Oct 30 21:06:26.680348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.680221    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"
Oct 30 21:06:26.680348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.680255    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v" event={"ID":"3002922a-ae83-42b9-8d51-08d606c5433f","Type":"ContainerDied","Data":"c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8"}
Oct 30 21:06:26.680348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.680301    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v" event={"ID":"3002922a-ae83-42b9-8d51-08d606c5433f","Type":"ContainerDied","Data":"57dc3545c2c9466250186df944c4764666f4e6225be3565b0620221ff4b9cfb0"}
Oct 30 21:06:26.680348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.680324    4810 scope.go:122] "RemoveContainer" containerID="c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8"
Oct 30 21:06:26.682530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.682508    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-703e17" event={"ID":"8a1bbe07-995e-44ab-8a35-24f0137e0f97","Type":"ContainerStarted","Data":"15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33"}
Oct 30 21:06:26.682646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.682545    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-703e17" event={"ID":"8a1bbe07-995e-44ab-8a35-24f0137e0f97","Type":"ContainerStarted","Data":"1d57985288ae54278de8a008322adb6d142eb232f3952074cb5177a9b3697d5b"}
Oct 30 21:06:26.682646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.682622    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7609-8b05dd" podUID="2179cd4c-1e2d-4b1f-8a10-fa1fda488373" containerName="test-curl-7609-8b05dd" containerID="cri-o://72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2" gracePeriod=2
Oct 30 21:06:26.702833 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.702781    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7609-703e17" podStartSLOduration=1.7027668710000001 podStartE2EDuration="1.702766871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:26.69799773 +0000 UTC m=+1753.189814269" watchObservedRunningTime="2026-10-30 21:06:26.702766871 +0000 UTC m=+1753.194583416"
Oct 30 21:06:26.727501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.727478    4810 scope.go:122] "RemoveContainer" containerID="c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8"
Oct 30 21:06:26.727780 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:06:26.727763    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8\": container with ID starting with c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8 not found: ID does not exist" containerID="c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8"
Oct 30 21:06:26.727867 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.727787    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8"} err="failed to get container status \"c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8\": rpc error: code = NotFound desc = could not find container \"c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8\": container with ID starting with c3ad953e978bd27f8ab2e055e271b2db95d3c46ee5f844f4f9437d68bf507df8 not found: ID does not exist"
Oct 30 21:06:26.737140 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.737109    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"]
Oct 30 21:06:26.743305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.743277    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-176db5/auth-test-model-176db5-kserve-764c8b88b9-bsq8v"]
Oct 30 21:06:26.889730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.889698    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-b45643"
Oct 30 21:06:26.950199 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.950169    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb4d4a26-009c-40ea-88ea-a3f43a678a0a-kube-api-access-vftrq\" (UniqueName: \"kubernetes.io/projected/cb4d4a26-009c-40ea-88ea-a3f43a678a0a-kube-api-access-vftrq\") pod \"cb4d4a26-009c-40ea-88ea-a3f43a678a0a\" (UID: \"cb4d4a26-009c-40ea-88ea-a3f43a678a0a\") "
Oct 30 21:06:26.952232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:26.952197    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4d4a26-009c-40ea-88ea-a3f43a678a0a-kube-api-access-vftrq" pod "cb4d4a26-009c-40ea-88ea-a3f43a678a0a" (UID: "cb4d4a26-009c-40ea-88ea-a3f43a678a0a"). InnerVolumeSpecName "kube-api-access-vftrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:27.052910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.052778    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vftrq\" (UniqueName: \"kubernetes.io/projected/cb4d4a26-009c-40ea-88ea-a3f43a678a0a-kube-api-access-vftrq\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:27.352725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.352066    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-703e17"]
Oct 30 21:06:27.356939 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.356908    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-703e17"]
Oct 30 21:06:27.509368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.509329    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-626c35"]
Oct 30 21:06:27.514371 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.514343    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8a1bbe07-995e-44ab-8a35-24f0137e0f97" containerName="test-curl-7609-703e17"
Oct 30 21:06:27.514371 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.514370    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1bbe07-995e-44ab-8a35-24f0137e0f97" containerName="test-curl-7609-703e17"
Oct 30 21:06:27.514573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.514400    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3002922a-ae83-42b9-8d51-08d606c5433f" containerName="main"
Oct 30 21:06:27.514573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.514407    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3002922a-ae83-42b9-8d51-08d606c5433f" containerName="main"
Oct 30 21:06:27.514682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.514607    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1bbe07-995e-44ab-8a35-24f0137e0f97" containerName="test-curl-7609-703e17"
Oct 30 21:06:27.514682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.514626    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3002922a-ae83-42b9-8d51-08d606c5433f" containerName="main"
Oct 30 21:06:27.518648 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.518623    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-626c35"]
Oct 30 21:06:27.518789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.518724    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-626c35"
Oct 30 21:06:27.559694 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.559663    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4zk\" (UniqueName: \"kubernetes.io/projected/dff08756-7e2d-4447-85a1-133b2877df96-kube-api-access-8c4zk\") pod \"test-curl-7609-626c35\" (UID: \"dff08756-7e2d-4447-85a1-133b2877df96\") " pod="opendatahub/test-curl-7609-626c35"
Oct 30 21:06:27.660867 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.660827    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4zk\" (UniqueName: \"kubernetes.io/projected/dff08756-7e2d-4447-85a1-133b2877df96-kube-api-access-8c4zk\") pod \"test-curl-7609-626c35\" (UID: \"dff08756-7e2d-4447-85a1-133b2877df96\") " pod="opendatahub/test-curl-7609-626c35"
Oct 30 21:06:27.663869 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.663834    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3002922a-ae83-42b9-8d51-08d606c5433f" path="/var/lib/kubelet/pods/3002922a-ae83-42b9-8d51-08d606c5433f/volumes"
Oct 30 21:06:27.664283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.664268    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb4d4a26-009c-40ea-88ea-a3f43a678a0a" path="/var/lib/kubelet/pods/cb4d4a26-009c-40ea-88ea-a3f43a678a0a/volumes"
Oct 30 21:06:27.669457 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.669434    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4zk\" (UniqueName: \"kubernetes.io/projected/dff08756-7e2d-4447-85a1-133b2877df96-kube-api-access-8c4zk\") pod \"test-curl-7609-626c35\" (UID: \"dff08756-7e2d-4447-85a1-133b2877df96\") " pod="opendatahub/test-curl-7609-626c35"
Oct 30 21:06:27.703705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.703658    4810 generic.go:356] "Generic (PLEG): container finished" podID="cb4d4a26-009c-40ea-88ea-a3f43a678a0a" containerID="64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609" exitCode=137
Oct 30 21:06:27.703971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.703840    4810 scope.go:122] "RemoveContainer" containerID="64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609"
Oct 30 21:06:27.704082 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.704063    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-b45643"
Oct 30 21:06:27.730115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.730094    4810 scope.go:122] "RemoveContainer" containerID="64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609"
Oct 30 21:06:27.730441 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:06:27.730423    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609\": container with ID starting with 64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609 not found: ID does not exist" containerID="64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609"
Oct 30 21:06:27.730526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.730449    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609"} err="failed to get container status \"64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609\": rpc error: code = NotFound desc = could not find container \"64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609\": container with ID starting with 64b88d727fbd52b9bac54dfad87dc17cbec83a6ed2d1aa6adb679c3afa816609 not found: ID does not exist"
Oct 30 21:06:27.835926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:27.835889    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-626c35"
Oct 30 21:06:28.097160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:28.097129    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"]
Oct 30 21:06:28.126007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:28.125975    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42e81d5-h54q8"]
Oct 30 21:06:28.162050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:28.162019    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-626c35"]
Oct 30 21:06:28.162338 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:06:28.162313    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddff08756_7e2d_4447_85a1_133b2877df96.slice/crio-676f9b9d9334cc46c467d55f96855dcdb6ca498189802c8267b8be9253818033 WatchSource:0}: Error finding container 676f9b9d9334cc46c467d55f96855dcdb6ca498189802c8267b8be9253818033: Status 404 returned error can't find the container with id 676f9b9d9334cc46c467d55f96855dcdb6ca498189802c8267b8be9253818033
Oct 30 21:06:28.722575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:28.722545    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-626c35" event={"ID":"dff08756-7e2d-4447-85a1-133b2877df96","Type":"ContainerStarted","Data":"cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad"}
Oct 30 21:06:28.723030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:28.722595    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-626c35" event={"ID":"dff08756-7e2d-4447-85a1-133b2877df96","Type":"ContainerStarted","Data":"676f9b9d9334cc46c467d55f96855dcdb6ca498189802c8267b8be9253818033"}
Oct 30 21:06:28.724122 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:28.724017    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7609-703e17" podUID="8a1bbe07-995e-44ab-8a35-24f0137e0f97" containerName="test-curl-7609-703e17" containerID="cri-o://15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33" gracePeriod=2
Oct 30 21:06:28.739484 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:28.739442    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7609-626c35" podStartSLOduration=1.7394286669999999 podStartE2EDuration="1.739428667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:28.736286443 +0000 UTC m=+1755.228102989" watchObservedRunningTime="2026-10-30 21:06:28.739428667 +0000 UTC m=+1755.231245191"
Oct 30 21:06:28.933193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:28.933170    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-8b05dd"
Oct 30 21:06:28.982112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:28.982042    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2179cd4c-1e2d-4b1f-8a10-fa1fda488373-kube-api-access-lzt2x\" (UniqueName: \"kubernetes.io/projected/2179cd4c-1e2d-4b1f-8a10-fa1fda488373-kube-api-access-lzt2x\") pod \"2179cd4c-1e2d-4b1f-8a10-fa1fda488373\" (UID: \"2179cd4c-1e2d-4b1f-8a10-fa1fda488373\") "
Oct 30 21:06:28.984282 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:28.984249    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2179cd4c-1e2d-4b1f-8a10-fa1fda488373-kube-api-access-lzt2x" pod "2179cd4c-1e2d-4b1f-8a10-fa1fda488373" (UID: "2179cd4c-1e2d-4b1f-8a10-fa1fda488373"). InnerVolumeSpecName "kube-api-access-lzt2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:29.084734 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.084705    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzt2x\" (UniqueName: \"kubernetes.io/projected/2179cd4c-1e2d-4b1f-8a10-fa1fda488373-kube-api-access-lzt2x\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:29.415797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.415764    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-626c35"]
Oct 30 21:06:29.421505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.421479    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-626c35"]
Oct 30 21:06:29.584902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.582791    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-4453f0"]
Oct 30 21:06:29.584902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.583847    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dff08756-7e2d-4447-85a1-133b2877df96" containerName="test-curl-7609-626c35"
Oct 30 21:06:29.584902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.583881    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff08756-7e2d-4447-85a1-133b2877df96" containerName="test-curl-7609-626c35"
Oct 30 21:06:29.584902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.584082    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff08756-7e2d-4447-85a1-133b2877df96" containerName="test-curl-7609-626c35"
Oct 30 21:06:29.587681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.587655    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-4453f0"
Oct 30 21:06:29.588123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.588099    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-4453f0"]
Oct 30 21:06:29.667698 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.667611    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2179cd4c-1e2d-4b1f-8a10-fa1fda488373" path="/var/lib/kubelet/pods/2179cd4c-1e2d-4b1f-8a10-fa1fda488373/volumes"
Oct 30 21:06:29.668495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.668468    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30cd1a7b-7650-4a0c-8b04-a11fd718032c" path="/var/lib/kubelet/pods/30cd1a7b-7650-4a0c-8b04-a11fd718032c/volumes"
Oct 30 21:06:29.694783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.694754    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvf2l\" (UniqueName: \"kubernetes.io/projected/e0e94ec2-4c44-47c0-8cca-2aed2c638960-kube-api-access-bvf2l\") pod \"test-curl-7609-4453f0\" (UID: \"e0e94ec2-4c44-47c0-8cca-2aed2c638960\") " pod="opendatahub/test-curl-7609-4453f0"
Oct 30 21:06:29.733422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.733392    4810 generic.go:356] "Generic (PLEG): container finished" podID="2179cd4c-1e2d-4b1f-8a10-fa1fda488373" containerID="72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2" exitCode=137
Oct 30 21:06:29.733805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.733461    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-8b05dd"
Oct 30 21:06:29.733805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.733485    4810 scope.go:122] "RemoveContainer" containerID="72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2"
Oct 30 21:06:29.743721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.743706    4810 scope.go:122] "RemoveContainer" containerID="72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2"
Oct 30 21:06:29.744035 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:06:29.744014    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2\": container with ID starting with 72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2 not found: ID does not exist" containerID="72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2"
Oct 30 21:06:29.744130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.744041    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2"} err="failed to get container status \"72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2\": rpc error: code = NotFound desc = could not find container \"72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2\": container with ID starting with 72b795b5f0ceed185887ff1070754418ee676214a207c560f0c8aa5a2e3d43e2 not found: ID does not exist"
Oct 30 21:06:29.796045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.796017    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvf2l\" (UniqueName: \"kubernetes.io/projected/e0e94ec2-4c44-47c0-8cca-2aed2c638960-kube-api-access-bvf2l\") pod \"test-curl-7609-4453f0\" (UID: \"e0e94ec2-4c44-47c0-8cca-2aed2c638960\") " pod="opendatahub/test-curl-7609-4453f0"
Oct 30 21:06:29.804659 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.804635    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvf2l\" (UniqueName: \"kubernetes.io/projected/e0e94ec2-4c44-47c0-8cca-2aed2c638960-kube-api-access-bvf2l\") pod \"test-curl-7609-4453f0\" (UID: \"e0e94ec2-4c44-47c0-8cca-2aed2c638960\") " pod="opendatahub/test-curl-7609-4453f0"
Oct 30 21:06:29.905537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:29.905498    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-4453f0"
Oct 30 21:06:30.187487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:30.187165    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-4453f0"]
Oct 30 21:06:30.189836 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:06:30.189796    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0e94ec2_4c44_47c0_8cca_2aed2c638960.slice/crio-5d3824fd4dd919cbea61513541de2391f0a685442a1364fac2187f71948ee618 WatchSource:0}: Error finding container 5d3824fd4dd919cbea61513541de2391f0a685442a1364fac2187f71948ee618: Status 404 returned error can't find the container with id 5d3824fd4dd919cbea61513541de2391f0a685442a1364fac2187f71948ee618
Oct 30 21:06:30.750076 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:30.750031    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-4453f0" event={"ID":"e0e94ec2-4c44-47c0-8cca-2aed2c638960","Type":"ContainerStarted","Data":"e2fdb4680259123b37041bb04f39c1ac45c705922bd68e6c03cd875324e7e293"}
Oct 30 21:06:30.750505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:30.750093    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-4453f0" event={"ID":"e0e94ec2-4c44-47c0-8cca-2aed2c638960","Type":"ContainerStarted","Data":"5d3824fd4dd919cbea61513541de2391f0a685442a1364fac2187f71948ee618"}
Oct 30 21:06:30.751444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:30.751394    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7609-626c35" podUID="dff08756-7e2d-4447-85a1-133b2877df96" containerName="test-curl-7609-626c35" containerID="cri-o://cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad" gracePeriod=2
Oct 30 21:06:30.764050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:30.763952    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7609-4453f0" podStartSLOduration=1.763933027 podStartE2EDuration="1.763933027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:30.76276076 +0000 UTC m=+1757.254577306" watchObservedRunningTime="2026-10-30 21:06:30.763933027 +0000 UTC m=+1757.255749573"
Oct 30 21:06:30.952441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:30.952415    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-703e17"
Oct 30 21:06:31.023976 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.023944    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1bbe07-995e-44ab-8a35-24f0137e0f97-kube-api-access-h8dv7\" (UniqueName: \"kubernetes.io/projected/8a1bbe07-995e-44ab-8a35-24f0137e0f97-kube-api-access-h8dv7\") pod \"8a1bbe07-995e-44ab-8a35-24f0137e0f97\" (UID: \"8a1bbe07-995e-44ab-8a35-24f0137e0f97\") "
Oct 30 21:06:31.026200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.026167    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1bbe07-995e-44ab-8a35-24f0137e0f97-kube-api-access-h8dv7" pod "8a1bbe07-995e-44ab-8a35-24f0137e0f97" (UID: "8a1bbe07-995e-44ab-8a35-24f0137e0f97"). InnerVolumeSpecName "kube-api-access-h8dv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:31.125815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.125718    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8dv7\" (UniqueName: \"kubernetes.io/projected/8a1bbe07-995e-44ab-8a35-24f0137e0f97-kube-api-access-h8dv7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:31.436867 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.436829    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-4453f0"]
Oct 30 21:06:31.441402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.441375    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-4453f0"]
Oct 30 21:06:31.589252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.589219    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-72666a"]
Oct 30 21:06:31.590208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.590174    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e0e94ec2-4c44-47c0-8cca-2aed2c638960" containerName="test-curl-7609-4453f0"
Oct 30 21:06:31.590208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.590200    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e94ec2-4c44-47c0-8cca-2aed2c638960" containerName="test-curl-7609-4453f0"
Oct 30 21:06:31.590419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.590374    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e94ec2-4c44-47c0-8cca-2aed2c638960" containerName="test-curl-7609-4453f0"
Oct 30 21:06:31.594432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.594410    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-72666a"
Oct 30 21:06:31.601482 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.601460    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-72666a"]
Oct 30 21:06:31.664473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.664445    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1bbe07-995e-44ab-8a35-24f0137e0f97" path="/var/lib/kubelet/pods/8a1bbe07-995e-44ab-8a35-24f0137e0f97/volumes"
Oct 30 21:06:31.734254 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.734187    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzxm\" (UniqueName: \"kubernetes.io/projected/95b6028b-2bba-46e1-9a8b-315bf176a453-kube-api-access-gmzxm\") pod \"test-curl-7609-72666a\" (UID: \"95b6028b-2bba-46e1-9a8b-315bf176a453\") " pod="opendatahub/test-curl-7609-72666a"
Oct 30 21:06:31.761268 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.761237    4810 generic.go:356] "Generic (PLEG): container finished" podID="8a1bbe07-995e-44ab-8a35-24f0137e0f97" containerID="15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33" exitCode=137
Oct 30 21:06:31.761671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.761296    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-703e17"
Oct 30 21:06:31.761671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.761331    4810 scope.go:122] "RemoveContainer" containerID="15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33"
Oct 30 21:06:31.772167 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.772149    4810 scope.go:122] "RemoveContainer" containerID="15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33"
Oct 30 21:06:31.772447 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:06:31.772430    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33\": container with ID starting with 15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33 not found: ID does not exist" containerID="15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33"
Oct 30 21:06:31.772501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.772458    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33"} err="failed to get container status \"15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33\": rpc error: code = NotFound desc = could not find container \"15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33\": container with ID starting with 15230e1bd9eff6bd19310cdef46fcfb78bf1bbd7a8f3d8afdb58ba1375672b33 not found: ID does not exist"
Oct 30 21:06:31.836337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.836304    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzxm\" (UniqueName: \"kubernetes.io/projected/95b6028b-2bba-46e1-9a8b-315bf176a453-kube-api-access-gmzxm\") pod \"test-curl-7609-72666a\" (UID: \"95b6028b-2bba-46e1-9a8b-315bf176a453\") " pod="opendatahub/test-curl-7609-72666a"
Oct 30 21:06:31.845506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.845472    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzxm\" (UniqueName: \"kubernetes.io/projected/95b6028b-2bba-46e1-9a8b-315bf176a453-kube-api-access-gmzxm\") pod \"test-curl-7609-72666a\" (UID: \"95b6028b-2bba-46e1-9a8b-315bf176a453\") " pod="opendatahub/test-curl-7609-72666a"
Oct 30 21:06:31.921961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:31.921928    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-72666a"
Oct 30 21:06:32.213520 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:32.213492    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-72666a"]
Oct 30 21:06:32.215431 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:06:32.215405    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95b6028b_2bba_46e1_9a8b_315bf176a453.slice/crio-3dca592199859daa641e53494170abc182e13b64ac8b7e184cc615b97c45a472 WatchSource:0}: Error finding container 3dca592199859daa641e53494170abc182e13b64ac8b7e184cc615b97c45a472: Status 404 returned error can't find the container with id 3dca592199859daa641e53494170abc182e13b64ac8b7e184cc615b97c45a472
Oct 30 21:06:32.803000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:32.802971    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-72666a" event={"ID":"95b6028b-2bba-46e1-9a8b-315bf176a453","Type":"ContainerStarted","Data":"def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7"}
Oct 30 21:06:32.803369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:32.803019    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-72666a" event={"ID":"95b6028b-2bba-46e1-9a8b-315bf176a453","Type":"ContainerStarted","Data":"3dca592199859daa641e53494170abc182e13b64ac8b7e184cc615b97c45a472"}
Oct 30 21:06:32.803430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:32.803380    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7609-4453f0" podUID="e0e94ec2-4c44-47c0-8cca-2aed2c638960" containerName="test-curl-7609-4453f0" containerID="cri-o://e2fdb4680259123b37041bb04f39c1ac45c705922bd68e6c03cd875324e7e293" gracePeriod=2
Oct 30 21:06:32.956670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:32.956646    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-626c35"
Oct 30 21:06:33.028918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.028798    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7609-72666a" podStartSLOduration=2.028784242 podStartE2EDuration="2.028784242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:32.823065161 +0000 UTC m=+1759.314881796" watchObservedRunningTime="2026-10-30 21:06:33.028784242 +0000 UTC m=+1759.520600787"
Oct 30 21:06:33.030820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.030795    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"]
Oct 30 21:06:33.034390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.034371    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-411319da-ggkrs"]
Oct 30 21:06:33.054278 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.054236    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff08756-7e2d-4447-85a1-133b2877df96-kube-api-access-8c4zk\" (UniqueName: \"kubernetes.io/projected/dff08756-7e2d-4447-85a1-133b2877df96-kube-api-access-8c4zk\") pod \"dff08756-7e2d-4447-85a1-133b2877df96\" (UID: \"dff08756-7e2d-4447-85a1-133b2877df96\") "
Oct 30 21:06:33.056506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.056478    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff08756-7e2d-4447-85a1-133b2877df96-kube-api-access-8c4zk" pod "dff08756-7e2d-4447-85a1-133b2877df96" (UID: "dff08756-7e2d-4447-85a1-133b2877df96"). InnerVolumeSpecName "kube-api-access-8c4zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:33.156830 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.156792    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4zk\" (UniqueName: \"kubernetes.io/projected/dff08756-7e2d-4447-85a1-133b2877df96-kube-api-access-8c4zk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:33.508882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.508724    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-72666a"]
Oct 30 21:06:33.531343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.531306    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-72666a"]
Oct 30 21:06:33.663741 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.663704    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff08756-7e2d-4447-85a1-133b2877df96" path="/var/lib/kubelet/pods/dff08756-7e2d-4447-85a1-133b2877df96/volumes"
Oct 30 21:06:33.664169 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.664154    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc6bb593-9e72-4a60-b93b-00649dfaf912" path="/var/lib/kubelet/pods/fc6bb593-9e72-4a60-b93b-00649dfaf912/volumes"
Oct 30 21:06:33.813004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.812895    4810 generic.go:356] "Generic (PLEG): container finished" podID="dff08756-7e2d-4447-85a1-133b2877df96" containerID="cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad" exitCode=137
Oct 30 21:06:33.813972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.813944    4810 scope.go:122] "RemoveContainer" containerID="cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad"
Oct 30 21:06:33.814132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.813953    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-626c35"
Oct 30 21:06:33.826170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.826149    4810 scope.go:122] "RemoveContainer" containerID="cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad"
Oct 30 21:06:33.826443 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:06:33.826424    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad\": container with ID starting with cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad not found: ID does not exist" containerID="cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad"
Oct 30 21:06:33.826485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:33.826454    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad"} err="failed to get container status \"cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad\": rpc error: code = NotFound desc = could not find container \"cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad\": container with ID starting with cddb44fb40cf89e78214d3da7658c3f4bb2cd7778bd24ef6823046302dd322ad not found: ID does not exist"
Oct 30 21:06:34.826911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:34.826848    4810 generic.go:356] "Generic (PLEG): container finished" podID="e0e94ec2-4c44-47c0-8cca-2aed2c638960" containerID="e2fdb4680259123b37041bb04f39c1ac45c705922bd68e6c03cd875324e7e293" exitCode=137
Oct 30 21:06:34.827274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:34.827139    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7609-72666a" podUID="95b6028b-2bba-46e1-9a8b-315bf176a453" containerName="test-curl-7609-72666a" containerID="cri-o://def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7" gracePeriod=2
Oct 30 21:06:35.007978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:35.007947    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-4453f0"
Oct 30 21:06:35.084833 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:35.084739    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e94ec2-4c44-47c0-8cca-2aed2c638960-kube-api-access-bvf2l\" (UniqueName: \"kubernetes.io/projected/e0e94ec2-4c44-47c0-8cca-2aed2c638960-kube-api-access-bvf2l\") pod \"e0e94ec2-4c44-47c0-8cca-2aed2c638960\" (UID: \"e0e94ec2-4c44-47c0-8cca-2aed2c638960\") "
Oct 30 21:06:35.087055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:35.087024    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e94ec2-4c44-47c0-8cca-2aed2c638960-kube-api-access-bvf2l" pod "e0e94ec2-4c44-47c0-8cca-2aed2c638960" (UID: "e0e94ec2-4c44-47c0-8cca-2aed2c638960"). InnerVolumeSpecName "kube-api-access-bvf2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:35.187289 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:35.187252    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvf2l\" (UniqueName: \"kubernetes.io/projected/e0e94ec2-4c44-47c0-8cca-2aed2c638960-kube-api-access-bvf2l\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:35.667475 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:35.667435    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e94ec2-4c44-47c0-8cca-2aed2c638960" path="/var/lib/kubelet/pods/e0e94ec2-4c44-47c0-8cca-2aed2c638960/volumes"
Oct 30 21:06:35.738878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:35.736333    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw" podUID="4a353a44-d648-4ada-9234-2314b56e4eaa" containerName="main" containerID="cri-o://fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449" gracePeriod=45
Oct 30 21:06:35.887135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:35.862732    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-4453f0"
Oct 30 21:06:35.887135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:35.862796    4810 scope.go:122] "RemoveContainer" containerID="e2fdb4680259123b37041bb04f39c1ac45c705922bd68e6c03cd875324e7e293"
Oct 30 21:06:36.100000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.099928    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"]
Oct 30 21:06:36.100431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.100400    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d0fd46-7z8q2"]
Oct 30 21:06:36.157583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.157557    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:06:36.215421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.215390    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-dshm\") pod \"4a353a44-d648-4ada-9234-2314b56e4eaa\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") "
Oct 30 21:06:36.215606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.215458    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a353a44-d648-4ada-9234-2314b56e4eaa-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a353a44-d648-4ada-9234-2314b56e4eaa-tls-certs\") pod \"4a353a44-d648-4ada-9234-2314b56e4eaa\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") "
Oct 30 21:06:36.215606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.215511    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-tmp-dir\") pod \"4a353a44-d648-4ada-9234-2314b56e4eaa\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") "
Oct 30 21:06:36.215606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.215547    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a353a44-d648-4ada-9234-2314b56e4eaa-kube-api-access-9xt9q\" (UniqueName: \"kubernetes.io/projected/4a353a44-d648-4ada-9234-2314b56e4eaa-kube-api-access-9xt9q\") pod \"4a353a44-d648-4ada-9234-2314b56e4eaa\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") "
Oct 30 21:06:36.215606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.215591    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-home\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-home\") pod \"4a353a44-d648-4ada-9234-2314b56e4eaa\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") "
Oct 30 21:06:36.215850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.215653    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-model-cache\") pod \"4a353a44-d648-4ada-9234-2314b56e4eaa\" (UID: \"4a353a44-d648-4ada-9234-2314b56e4eaa\") "
Oct 30 21:06:36.216348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.216310    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-tmp-dir" pod "4a353a44-d648-4ada-9234-2314b56e4eaa" (UID: "4a353a44-d648-4ada-9234-2314b56e4eaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:36.216847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.216356    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-model-cache" pod "4a353a44-d648-4ada-9234-2314b56e4eaa" (UID: "4a353a44-d648-4ada-9234-2314b56e4eaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:36.217023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.216786    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-home" pod "4a353a44-d648-4ada-9234-2314b56e4eaa" (UID: "4a353a44-d648-4ada-9234-2314b56e4eaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:36.219005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.218982    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a353a44-d648-4ada-9234-2314b56e4eaa-kube-api-access-9xt9q" pod "4a353a44-d648-4ada-9234-2314b56e4eaa" (UID: "4a353a44-d648-4ada-9234-2314b56e4eaa"). InnerVolumeSpecName "kube-api-access-9xt9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:36.240114 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.239324    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a353a44-d648-4ada-9234-2314b56e4eaa-tls-certs" pod "4a353a44-d648-4ada-9234-2314b56e4eaa" (UID: "4a353a44-d648-4ada-9234-2314b56e4eaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:36.243880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.241140    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-dshm" pod "4a353a44-d648-4ada-9234-2314b56e4eaa" (UID: "4a353a44-d648-4ada-9234-2314b56e4eaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:36.317048 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.317015    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:36.317048 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.317044    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a353a44-d648-4ada-9234-2314b56e4eaa-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:36.317250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.317072    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:36.317250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.317085    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xt9q\" (UniqueName: \"kubernetes.io/projected/4a353a44-d648-4ada-9234-2314b56e4eaa-kube-api-access-9xt9q\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:36.317250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.317095    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:36.317250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.317104    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a353a44-d648-4ada-9234-2314b56e4eaa-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:36.874821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.874788    4810 generic.go:356] "Generic (PLEG): container finished" podID="4a353a44-d648-4ada-9234-2314b56e4eaa" containerID="fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449" exitCode=0
Oct 30 21:06:36.875022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.874997    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"
Oct 30 21:06:36.875097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.875056    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw" event={"ID":"4a353a44-d648-4ada-9234-2314b56e4eaa","Type":"ContainerDied","Data":"fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449"}
Oct 30 21:06:36.875162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.875098    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw" event={"ID":"4a353a44-d648-4ada-9234-2314b56e4eaa","Type":"ContainerDied","Data":"3e69465885068cbe5e87210691d44101402dbf214729cface7a157019a3458b3"}
Oct 30 21:06:36.875162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.875123    4810 scope.go:122] "RemoveContainer" containerID="fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449"
Oct 30 21:06:36.902390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.902368    4810 scope.go:122] "RemoveContainer" containerID="fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449"
Oct 30 21:06:36.902789 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:06:36.902757    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449\": container with ID starting with fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449 not found: ID does not exist" containerID="fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449"
Oct 30 21:06:36.902853 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.902813    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449"} err="failed to get container status \"fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449\": rpc error: code = NotFound desc = could not find container \"fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449\": container with ID starting with fb74b4527c7a355226e6834a7138c4ce6f515b81ddcf2976d2eea7b57d923449 not found: ID does not exist"
Oct 30 21:06:36.924895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.922963    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"]
Oct 30 21:06:36.927047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:36.927026    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-74f9f4/auth-test-model-74f9f4-kserve-5bcd6f6f45-b2qzw"]
Oct 30 21:06:37.095242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.089083    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-72666a"
Oct 30 21:06:37.233432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.233396    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b6028b-2bba-46e1-9a8b-315bf176a453-kube-api-access-gmzxm\" (UniqueName: \"kubernetes.io/projected/95b6028b-2bba-46e1-9a8b-315bf176a453-kube-api-access-gmzxm\") pod \"95b6028b-2bba-46e1-9a8b-315bf176a453\" (UID: \"95b6028b-2bba-46e1-9a8b-315bf176a453\") "
Oct 30 21:06:37.236215 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.236185    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b6028b-2bba-46e1-9a8b-315bf176a453-kube-api-access-gmzxm" pod "95b6028b-2bba-46e1-9a8b-315bf176a453" (UID: "95b6028b-2bba-46e1-9a8b-315bf176a453"). InnerVolumeSpecName "kube-api-access-gmzxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:37.337128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.337090    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmzxm\" (UniqueName: \"kubernetes.io/projected/95b6028b-2bba-46e1-9a8b-315bf176a453-kube-api-access-gmzxm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:37.665201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.665162    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a353a44-d648-4ada-9234-2314b56e4eaa" path="/var/lib/kubelet/pods/4a353a44-d648-4ada-9234-2314b56e4eaa/volumes"
Oct 30 21:06:37.665910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.665885    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b6028b-2bba-46e1-9a8b-315bf176a453" path="/var/lib/kubelet/pods/95b6028b-2bba-46e1-9a8b-315bf176a453/volumes"
Oct 30 21:06:37.666737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.666711    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a541be3f-37eb-4ca3-970a-7032c68188dc" path="/var/lib/kubelet/pods/a541be3f-37eb-4ca3-970a-7032c68188dc/volumes"
Oct 30 21:06:37.885440 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.885398    4810 generic.go:356] "Generic (PLEG): container finished" podID="95b6028b-2bba-46e1-9a8b-315bf176a453" containerID="def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7" exitCode=137
Oct 30 21:06:37.885623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.885454    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-72666a"
Oct 30 21:06:37.885682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.885636    4810 scope.go:122] "RemoveContainer" containerID="def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7"
Oct 30 21:06:37.898272 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.898249    4810 scope.go:122] "RemoveContainer" containerID="def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7"
Oct 30 21:06:37.898570 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:06:37.898545    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7\": container with ID starting with def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7 not found: ID does not exist" containerID="def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7"
Oct 30 21:06:37.898664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:37.898581    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7"} err="failed to get container status \"def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7\": rpc error: code = NotFound desc = could not find container \"def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7\": container with ID starting with def12095dae7e81f6539e6f3a341b6e958fc1c8548f6c5556e2c17ee5576f3a7 not found: ID does not exist"
Oct 30 21:06:40.785423 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.785387    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"]
Oct 30 21:06:40.786093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.786078    4810 cpu_manager.go:414] "containerMap: removing container" podUID="95b6028b-2bba-46e1-9a8b-315bf176a453" containerName="test-curl-7609-72666a"
Oct 30 21:06:40.786157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.786095    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b6028b-2bba-46e1-9a8b-315bf176a453" containerName="test-curl-7609-72666a"
Oct 30 21:06:40.786157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.786127    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4a353a44-d648-4ada-9234-2314b56e4eaa" containerName="main"
Oct 30 21:06:40.786157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.786133    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a353a44-d648-4ada-9234-2314b56e4eaa" containerName="main"
Oct 30 21:06:40.786256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.786235    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b6028b-2bba-46e1-9a8b-315bf176a453" containerName="test-curl-7609-72666a"
Oct 30 21:06:40.786256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.786249    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a353a44-d648-4ada-9234-2314b56e4eaa" containerName="main"
Oct 30 21:06:40.791333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.791312    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:06:40.799750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.799718    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"]
Oct 30 21:06:40.804414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.804323    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-345bc4\""
Oct 30 21:06:40.814333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.814308    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"]
Oct 30 21:06:40.814455 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.814352    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"]
Oct 30 21:06:40.814455 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.814436    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:06:40.821711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.820316    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-345bc4-dockercfg-tw8k8\""
Oct 30 21:06:40.910893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.908718    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k4fd\" (UniqueName: \"kubernetes.io/projected/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-kube-api-access-2k4fd\") pod \"payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4\" (UID: \"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:06:40.910893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.908780    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4\" (UID: \"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:06:40.910893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.908967    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-345bc4-7555945897-88pml\" (UID: \"ebf45ae6-8684-4f2d-86a2-fa2d6930f786\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:06:40.910893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:40.909014    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgqvx\" (UniqueName: \"kubernetes.io/projected/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-kube-api-access-tgqvx\") pod \"payload-processing-e2e-extmod-a-345bc4-7555945897-88pml\" (UID: \"ebf45ae6-8684-4f2d-86a2-fa2d6930f786\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:06:41.018246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.018203    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-345bc4-7555945897-88pml\" (UID: \"ebf45ae6-8684-4f2d-86a2-fa2d6930f786\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:06:41.018451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.018295    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgqvx\" (UniqueName: \"kubernetes.io/projected/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-kube-api-access-tgqvx\") pod \"payload-processing-e2e-extmod-a-345bc4-7555945897-88pml\" (UID: \"ebf45ae6-8684-4f2d-86a2-fa2d6930f786\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:06:41.018552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.018529    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k4fd\" (UniqueName: \"kubernetes.io/projected/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-kube-api-access-2k4fd\") pod \"payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4\" (UID: \"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:06:41.018627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.018595    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4\" (UID: \"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:06:41.019287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.019261    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-345bc4-7555945897-88pml\" (UID: \"ebf45ae6-8684-4f2d-86a2-fa2d6930f786\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:06:41.019951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.019923    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4\" (UID: \"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:06:41.030397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.030367    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k4fd\" (UniqueName: \"kubernetes.io/projected/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-kube-api-access-2k4fd\") pod \"payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4\" (UID: \"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:06:41.033457 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.032491    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgqvx\" (UniqueName: \"kubernetes.io/projected/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-kube-api-access-tgqvx\") pod \"payload-processing-e2e-extmod-a-345bc4-7555945897-88pml\" (UID: \"ebf45ae6-8684-4f2d-86a2-fa2d6930f786\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:06:41.037238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.036914    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"]
Oct 30 21:06:41.043243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.043221    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-169cbb63-t8pzv"]
Oct 30 21:06:41.113345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.113313    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:06:41.134784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.134757    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:06:41.449206 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:06:41.448999    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf45ae6_8684_4f2d_86a2_fa2d6930f786.slice/crio-58ed79ac95889ad58553e4bf55924c11eb18928aa69f1bf0aed2774ab34c2307 WatchSource:0}: Error finding container 58ed79ac95889ad58553e4bf55924c11eb18928aa69f1bf0aed2774ab34c2307: Status 404 returned error can't find the container with id 58ed79ac95889ad58553e4bf55924c11eb18928aa69f1bf0aed2774ab34c2307
Oct 30 21:06:41.450961 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:06:41.450905    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6a9d0f_fc5a_490a_a0f1_a710e920ff2e.slice/crio-7c8f23c2eacb9a890d58eda219fe0c24b18320fdcd06adf63235c70e6b70b930 WatchSource:0}: Error finding container 7c8f23c2eacb9a890d58eda219fe0c24b18320fdcd06adf63235c70e6b70b930: Status 404 returned error can't find the container with id 7c8f23c2eacb9a890d58eda219fe0c24b18320fdcd06adf63235c70e6b70b930
Oct 30 21:06:41.451966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.451902    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"]
Oct 30 21:06:41.457263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.457239    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"]
Oct 30 21:06:41.663793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.663763    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ce23a0-3974-4c00-b4a7-b0920d00748d" path="/var/lib/kubelet/pods/17ce23a0-3974-4c00-b4a7-b0920d00748d/volumes"
Oct 30 21:06:41.934506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.934472    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4" event={"ID":"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e","Type":"ContainerStarted","Data":"692a21474f02aac3d63a261cf803282601e2f35886cd6b0e2a9694dd2b7c353d"}
Oct 30 21:06:41.934945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.934538    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:06:41.934945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.934556    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4" event={"ID":"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e","Type":"ContainerStarted","Data":"7c8f23c2eacb9a890d58eda219fe0c24b18320fdcd06adf63235c70e6b70b930"}
Oct 30 21:06:41.936253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.936227    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml" event={"ID":"ebf45ae6-8684-4f2d-86a2-fa2d6930f786","Type":"ContainerStarted","Data":"037e07ba7ccc196e52b22b4b9613be23034049b0e4791ccde94b54a5a1d26932"}
Oct 30 21:06:41.936374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.936276    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml" event={"ID":"ebf45ae6-8684-4f2d-86a2-fa2d6930f786","Type":"ContainerStarted","Data":"58ed79ac95889ad58553e4bf55924c11eb18928aa69f1bf0aed2774ab34c2307"}
Oct 30 21:06:41.936374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.936298    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:06:41.958573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.958531    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4" podStartSLOduration=1.9585191480000002 podStartE2EDuration="1.958519148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:41.956743876 +0000 UTC m=+1768.448560420" watchObservedRunningTime="2026-10-30 21:06:41.958519148 +0000 UTC m=+1768.450335702"
Oct 30 21:06:41.976911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:41.976871    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml" podStartSLOduration=1.9768460270000001 podStartE2EDuration="1.976846027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:41.974296205 +0000 UTC m=+1768.466112751" watchObservedRunningTime="2026-10-30 21:06:41.976846027 +0000 UTC m=+1768.468662572"
Oct 30 21:06:48.027190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:48.027144    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"]
Oct 30 21:06:48.032797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:48.032764    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ef2303-vzcw4"]
Oct 30 21:06:49.031523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:49.031493    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"]
Oct 30 21:06:49.034920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:49.034894    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaeb7edb-7jpb9"]
Oct 30 21:06:49.663472 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:49.663444    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c6b938-5c3f-4222-89cb-244426dc086e" path="/var/lib/kubelet/pods/b8c6b938-5c3f-4222-89cb-244426dc086e/volumes"
Oct 30 21:06:49.663945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:49.663930    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde" path="/var/lib/kubelet/pods/f0ece89a-a4bb-4bf8-b8b0-4e8655ec5dde/volumes"
Oct 30 21:06:52.960665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:52.960634    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:06:52.966039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:06:52.966011    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:06:55.163144 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:06:55.163107    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:07:01.029645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:01.029614    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"]
Oct 30 21:07:01.037814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:01.037790    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04ce478b-lx5mw"]
Oct 30 21:07:01.663221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:01.663189    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95d7998-c053-4219-bb4d-b8e27b03dc9e" path="/var/lib/kubelet/pods/c95d7998-c053-4219-bb4d-b8e27b03dc9e/volumes"
Oct 30 21:07:04.026207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:04.026165    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"]
Oct 30 21:07:04.029507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:04.029485    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fdef763-l7ff7"]
Oct 30 21:07:05.662616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:05.662584    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceec1e25-2da9-4a0d-9737-77e6b01b19a3" path="/var/lib/kubelet/pods/ceec1e25-2da9-4a0d-9737-77e6b01b19a3/volumes"
Oct 30 21:07:06.453433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:06.453400    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:07:09.010318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.010288    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"]
Oct 30 21:07:09.015144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.015119    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.024913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.024845    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:07:09.025073 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.025038    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:07:09.026645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.026618    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"]
Oct 30 21:07:09.190389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.190346    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.190595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.190413    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.190669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.190640    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkxk\" (UniqueName: \"kubernetes.io/projected/a5a872ae-eb38-47a4-8410-b9928f1d94be-kube-api-access-8gkxk\") pod \"maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.291691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.291598    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.291691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.291644    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.291973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.291787    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkxk\" (UniqueName: \"kubernetes.io/projected/a5a872ae-eb38-47a4-8410-b9928f1d94be-kube-api-access-8gkxk\") pod \"maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.292392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.292369    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.294051 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.294021    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.301722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.301690    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkxk\" (UniqueName: \"kubernetes.io/projected/a5a872ae-eb38-47a4-8410-b9928f1d94be-kube-api-access-8gkxk\") pod \"maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.346616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.346596    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:09.392989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.392940    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"]
Oct 30 21:07:09.413076 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.409921    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"]
Oct 30 21:07:09.413076 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.410037    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.595664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.595586    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09750a7-k65th\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.595664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.595650    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tbs\" (UniqueName: \"kubernetes.io/projected/57c15b3a-53dd-4554-8141-b8a1ec58727e-kube-api-access-28tbs\") pod \"maas-api-revoke-sub-e09750a7-k65th\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.596001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.595984    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09750a7-k65th\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.683209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.683183    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"]
Oct 30 21:07:09.684716 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:09.684683    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a872ae_eb38_47a4_8410_b9928f1d94be.slice/crio-bc5a73f744420a3e5099e59946e6bcdec9f3af84c32c6a807cb5a9d5faba2471 WatchSource:0}: Error finding container bc5a73f744420a3e5099e59946e6bcdec9f3af84c32c6a807cb5a9d5faba2471: Status 404 returned error can't find the container with id bc5a73f744420a3e5099e59946e6bcdec9f3af84c32c6a807cb5a9d5faba2471
Oct 30 21:07:09.697267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.697241    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09750a7-k65th\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.697359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.697307    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09750a7-k65th\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.697419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.697359    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tbs\" (UniqueName: \"kubernetes.io/projected/57c15b3a-53dd-4554-8141-b8a1ec58727e-kube-api-access-28tbs\") pod \"maas-api-revoke-sub-e09750a7-k65th\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.698028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.698008    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09750a7-k65th\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.700171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.700149    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09750a7-k65th\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.709615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.709590    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tbs\" (UniqueName: \"kubernetes.io/projected/57c15b3a-53dd-4554-8141-b8a1ec58727e-kube-api-access-28tbs\") pod \"maas-api-revoke-sub-e09750a7-k65th\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.742257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.742237    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:09.809758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.809731    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"]
Oct 30 21:07:09.814338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.814307    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:09.830871 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:09.830124    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"]
Oct 30 21:07:10.004579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.004545    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5775d53-dnfcc\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:10.004768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.004746    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5775d53-dnfcc\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:10.004904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.004889    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxpk\" (UniqueName: \"kubernetes.io/projected/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-kube-api-access-2wxpk\") pod \"maas-api-revoke-sub-c5775d53-dnfcc\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:10.043500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.040168    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"]
Oct 30 21:07:10.106386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.106358    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5775d53-dnfcc\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:10.106522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.106459    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxpk\" (UniqueName: \"kubernetes.io/projected/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-kube-api-access-2wxpk\") pod \"maas-api-revoke-sub-c5775d53-dnfcc\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:10.106586 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.106545    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5775d53-dnfcc\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:10.107126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.107107    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5775d53-dnfcc\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:10.108654 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.108635    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5775d53-dnfcc\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:10.115392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.115365    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxpk\" (UniqueName: \"kubernetes.io/projected/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-kube-api-access-2wxpk\") pod \"maas-api-revoke-sub-c5775d53-dnfcc\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:10.143369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.143320    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:10.243494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.242587    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th" event={"ID":"57c15b3a-53dd-4554-8141-b8a1ec58727e","Type":"ContainerStarted","Data":"6d183d599037650dbf0b81eed82765c0bff5f5f733590cb1ffcc80c506899fdf"}
Oct 30 21:07:10.245214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.244299    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2" event={"ID":"a5a872ae-eb38-47a4-8410-b9928f1d94be","Type":"ContainerStarted","Data":"64e462cd454bed25c42a286f07daba1ab93d8860075aa12df16326711aca4b07"}
Oct 30 21:07:10.245214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.244340    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2" event={"ID":"a5a872ae-eb38-47a4-8410-b9928f1d94be","Type":"ContainerStarted","Data":"bc5a73f744420a3e5099e59946e6bcdec9f3af84c32c6a807cb5a9d5faba2471"}
Oct 30 21:07:10.269324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.269272    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2" podStartSLOduration=2.269255057 podStartE2EDuration="2.269255057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:10.266114109 +0000 UTC m=+1796.757930645" watchObservedRunningTime="2026-10-30 21:07:10.269255057 +0000 UTC m=+1796.761071601"
Oct 30 21:07:10.435433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:10.435400    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"]
Oct 30 21:07:10.436254 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:10.436224    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354df1b9_f265_40bf_bb6a_0e3bffa8dbb6.slice/crio-89c044da99c807b8056b509267eb892ed4cc7a6cde1468a23211350ac69ff3de WatchSource:0}: Error finding container 89c044da99c807b8056b509267eb892ed4cc7a6cde1468a23211350ac69ff3de: Status 404 returned error can't find the container with id 89c044da99c807b8056b509267eb892ed4cc7a6cde1468a23211350ac69ff3de
Oct 30 21:07:11.255312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:11.255197    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc" event={"ID":"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6","Type":"ContainerStarted","Data":"fab35b2d43cdaf460baabf8189963e0f9603a6cd3a55cd0492a1917c5d8adca8"}
Oct 30 21:07:11.255312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:11.255273    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc" event={"ID":"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6","Type":"ContainerStarted","Data":"89c044da99c807b8056b509267eb892ed4cc7a6cde1468a23211350ac69ff3de"}
Oct 30 21:07:11.258100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:11.258073    4810 generic.go:356] "Generic (PLEG): container finished" podID="57c15b3a-53dd-4554-8141-b8a1ec58727e" containerID="cd6d4aa7bc95d87a0a0c0577a14967c3df7342933891ab1c207c9900a9d85dcd" exitCode=0
Oct 30 21:07:11.258235 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:11.258161    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th" event={"ID":"57c15b3a-53dd-4554-8141-b8a1ec58727e","Type":"ContainerDied","Data":"cd6d4aa7bc95d87a0a0c0577a14967c3df7342933891ab1c207c9900a9d85dcd"}
Oct 30 21:07:11.260063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:11.260034    4810 generic.go:356] "Generic (PLEG): container finished" podID="a5a872ae-eb38-47a4-8410-b9928f1d94be" containerID="64e462cd454bed25c42a286f07daba1ab93d8860075aa12df16326711aca4b07" exitCode=0
Oct 30 21:07:11.260195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:11.260124    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2" event={"ID":"a5a872ae-eb38-47a4-8410-b9928f1d94be","Type":"ContainerDied","Data":"64e462cd454bed25c42a286f07daba1ab93d8860075aa12df16326711aca4b07"}
Oct 30 21:07:11.286879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:11.283225    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc" podStartSLOduration=2.283210343 podStartE2EDuration="2.283210343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:11.279474432 +0000 UTC m=+1797.771291038" watchObservedRunningTime="2026-10-30 21:07:11.283210343 +0000 UTC m=+1797.775026892"
Oct 30 21:07:12.270176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.270141    4810 generic.go:356] "Generic (PLEG): container finished" podID="354df1b9-f265-40bf-bb6a-0e3bffa8dbb6" containerID="fab35b2d43cdaf460baabf8189963e0f9603a6cd3a55cd0492a1917c5d8adca8" exitCode=0
Oct 30 21:07:12.270653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.270256    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc" event={"ID":"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6","Type":"ContainerDied","Data":"fab35b2d43cdaf460baabf8189963e0f9603a6cd3a55cd0492a1917c5d8adca8"}
Oct 30 21:07:12.569600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.569572    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:12.633469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.633174    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:12.663551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.663522    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-openshift-ca\") pod \"57c15b3a-53dd-4554-8141-b8a1ec58727e\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") "
Oct 30 21:07:12.663730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.663603    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-openshift-ca\") pod \"a5a872ae-eb38-47a4-8410-b9928f1d94be\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") "
Oct 30 21:07:12.663730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.663685    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-xks-ca\") pod \"a5a872ae-eb38-47a4-8410-b9928f1d94be\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") "
Oct 30 21:07:12.663889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.663782    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c15b3a-53dd-4554-8141-b8a1ec58727e-kube-api-access-28tbs\" (UniqueName: \"kubernetes.io/projected/57c15b3a-53dd-4554-8141-b8a1ec58727e-kube-api-access-28tbs\") pod \"57c15b3a-53dd-4554-8141-b8a1ec58727e\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") "
Oct 30 21:07:12.663889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.663840    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-xks-ca\") pod \"57c15b3a-53dd-4554-8141-b8a1ec58727e\" (UID: \"57c15b3a-53dd-4554-8141-b8a1ec58727e\") "
Oct 30 21:07:12.664010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.663973    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a872ae-eb38-47a4-8410-b9928f1d94be-kube-api-access-8gkxk\" (UniqueName: \"kubernetes.io/projected/a5a872ae-eb38-47a4-8410-b9928f1d94be-kube-api-access-8gkxk\") pod \"a5a872ae-eb38-47a4-8410-b9928f1d94be\" (UID: \"a5a872ae-eb38-47a4-8410-b9928f1d94be\") "
Oct 30 21:07:12.664010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.663986    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-openshift-ca" pod "57c15b3a-53dd-4554-8141-b8a1ec58727e" (UID: "57c15b3a-53dd-4554-8141-b8a1ec58727e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:12.664545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.664518    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-openshift-ca" pod "a5a872ae-eb38-47a4-8410-b9928f1d94be" (UID: "a5a872ae-eb38-47a4-8410-b9928f1d94be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:12.664545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.664530    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:12.668178 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.668141    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a872ae-eb38-47a4-8410-b9928f1d94be-kube-api-access-8gkxk" pod "a5a872ae-eb38-47a4-8410-b9928f1d94be" (UID: "a5a872ae-eb38-47a4-8410-b9928f1d94be"). InnerVolumeSpecName "kube-api-access-8gkxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:12.682775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.682595    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-xks-ca" pod "57c15b3a-53dd-4554-8141-b8a1ec58727e" (UID: "57c15b3a-53dd-4554-8141-b8a1ec58727e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:12.682775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.682714    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c15b3a-53dd-4554-8141-b8a1ec58727e-kube-api-access-28tbs" pod "57c15b3a-53dd-4554-8141-b8a1ec58727e" (UID: "57c15b3a-53dd-4554-8141-b8a1ec58727e"). InnerVolumeSpecName "kube-api-access-28tbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:12.687148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.687127    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-xks-ca" pod "a5a872ae-eb38-47a4-8410-b9928f1d94be" (UID: "a5a872ae-eb38-47a4-8410-b9928f1d94be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:12.766425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.766397    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:12.766425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.766421    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a872ae-eb38-47a4-8410-b9928f1d94be-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:12.766632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.766433    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28tbs\" (UniqueName: \"kubernetes.io/projected/57c15b3a-53dd-4554-8141-b8a1ec58727e-kube-api-access-28tbs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:12.766632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.766442    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c15b3a-53dd-4554-8141-b8a1ec58727e-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:12.766632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:12.766451    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gkxk\" (UniqueName: \"kubernetes.io/projected/a5a872ae-eb38-47a4-8410-b9928f1d94be-kube-api-access-8gkxk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:13.298881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.283128    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"
Oct 30 21:07:13.298881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.283162    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th" event={"ID":"57c15b3a-53dd-4554-8141-b8a1ec58727e","Type":"ContainerDied","Data":"6d183d599037650dbf0b81eed82765c0bff5f5f733590cb1ffcc80c506899fdf"}
Oct 30 21:07:13.298881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.283223    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d183d599037650dbf0b81eed82765c0bff5f5f733590cb1ffcc80c506899fdf"
Oct 30 21:07:13.298881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.287028    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2" event={"ID":"a5a872ae-eb38-47a4-8410-b9928f1d94be","Type":"ContainerDied","Data":"bc5a73f744420a3e5099e59946e6bcdec9f3af84c32c6a807cb5a9d5faba2471"}
Oct 30 21:07:13.298881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.287049    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"
Oct 30 21:07:13.298881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.287072    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc5a73f744420a3e5099e59946e6bcdec9f3af84c32c6a807cb5a9d5faba2471"
Oct 30 21:07:13.466996 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.466975    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:13.489616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.488391    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-xks-ca\") pod \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") "
Oct 30 21:07:13.489616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.488450    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-openshift-ca\") pod \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") "
Oct 30 21:07:13.489616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.488578    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-kube-api-access-2wxpk\" (UniqueName: \"kubernetes.io/projected/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-kube-api-access-2wxpk\") pod \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\" (UID: \"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6\") "
Oct 30 21:07:13.490633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.490608    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-openshift-ca" pod "354df1b9-f265-40bf-bb6a-0e3bffa8dbb6" (UID: "354df1b9-f265-40bf-bb6a-0e3bffa8dbb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:13.500422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.500383    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-kube-api-access-2wxpk" pod "354df1b9-f265-40bf-bb6a-0e3bffa8dbb6" (UID: "354df1b9-f265-40bf-bb6a-0e3bffa8dbb6"). InnerVolumeSpecName "kube-api-access-2wxpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:13.508311 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.508273    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-xks-ca" pod "354df1b9-f265-40bf-bb6a-0e3bffa8dbb6" (UID: "354df1b9-f265-40bf-bb6a-0e3bffa8dbb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:13.590374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.590294    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:13.590374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.590319    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:13.590374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:13.590331    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wxpk\" (UniqueName: \"kubernetes.io/projected/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6-kube-api-access-2wxpk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:14.028364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:14.028333    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"]
Oct 30 21:07:14.032160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:14.032138    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-347fecc2-48prq"]
Oct 30 21:07:14.296157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:14.296069    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"
Oct 30 21:07:14.296157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:14.296082    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc" event={"ID":"354df1b9-f265-40bf-bb6a-0e3bffa8dbb6","Type":"ContainerDied","Data":"89c044da99c807b8056b509267eb892ed4cc7a6cde1468a23211350ac69ff3de"}
Oct 30 21:07:14.296384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:14.296164    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89c044da99c807b8056b509267eb892ed4cc7a6cde1468a23211350ac69ff3de"
Oct 30 21:07:14.372584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:14.372552    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:07:14.407118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:14.407098    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:07:15.662904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.662875    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2" path="/var/lib/kubelet/pods/080fb2aa-f6dd-4f74-ab47-9bdb87eb70d2/volumes"
Oct 30 21:07:15.732879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.732015    4810 scope.go:122] "RemoveContainer" containerID="7c9aaef4b7926e2679b94b3827ddc7fd8d64d1dfdf90876b61f25227fdaaff6c"
Oct 30 21:07:15.751320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.751299    4810 scope.go:122] "RemoveContainer" containerID="b6150e1600eaa7b749cb64dd4fce2e0376b736ffbe342cbd648b993550739433"
Oct 30 21:07:15.764621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.764586    4810 scope.go:122] "RemoveContainer" containerID="4b97f058926bb4c8493c3a1ecbf220c26927bc18bdbfc4d380febe3543b4af2f"
Oct 30 21:07:15.777782 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.777757    4810 scope.go:122] "RemoveContainer" containerID="08e4fee36feaf8d9ea80db8ce22bc608a7e222077d5a208a0116a25a1255321d"
Oct 30 21:07:15.792931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.792911    4810 scope.go:122] "RemoveContainer" containerID="04198f2bd4a7f36be78136dfb2054e2dc8274e886e03e2292a91d325ea88b6af"
Oct 30 21:07:15.804715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.804694    4810 scope.go:122] "RemoveContainer" containerID="057b7761e48df49e1c67014a842db7cfba39f760ebb5e9f57b197f4813866f5f"
Oct 30 21:07:15.815613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.815592    4810 scope.go:122] "RemoveContainer" containerID="34fa5da94cea12b47b690bd597b310177a58b7c096acf925e7c53bca83ed249a"
Oct 30 21:07:15.826426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.826398    4810 scope.go:122] "RemoveContainer" containerID="c9af7784341c8292439029d7dd8dc4c001662d32be326d23b807a60da8d91d8d"
Oct 30 21:07:15.835394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.835374    4810 scope.go:122] "RemoveContainer" containerID="c70ba651faa79618f3c13ee2806027d494e99852f6b695d7f02c5adf401c9da4"
Oct 30 21:07:15.845498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.845478    4810 scope.go:122] "RemoveContainer" containerID="36417ba24c9f6e029f3e49c5860678d8ab2e4b3e1cdd46327579d2a9bcb42464"
Oct 30 21:07:15.854376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.854361    4810 scope.go:122] "RemoveContainer" containerID="490d4a68cdaae90487f2116976a7a071c88e527f56ef0e5f60ec44d16b859f56"
Oct 30 21:07:15.863295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.863279    4810 scope.go:122] "RemoveContainer" containerID="ff792dff393d7f62760b7dfec6f44767483ea737b3206082759bdae103f8b5b2"
Oct 30 21:07:15.872814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:15.872794    4810 scope.go:122] "RemoveContainer" containerID="1991e007982b14b2ee9a4254290e9d8755ffbd8158ed22512994d31bd89f4e90"
Oct 30 21:07:16.474883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.474829    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"]
Oct 30 21:07:16.475896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.475868    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a5a872ae-eb38-47a4-8410-b9928f1d94be" containerName="revoke-keys"
Oct 30 21:07:16.475896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.475894    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a872ae-eb38-47a4-8410-b9928f1d94be" containerName="revoke-keys"
Oct 30 21:07:16.476043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.475964    4810 cpu_manager.go:414] "containerMap: removing container" podUID="354df1b9-f265-40bf-bb6a-0e3bffa8dbb6" containerName="revoke-keys"
Oct 30 21:07:16.476043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.475973    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354df1b9-f265-40bf-bb6a-0e3bffa8dbb6" containerName="revoke-keys"
Oct 30 21:07:16.476043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.476012    4810 cpu_manager.go:414] "containerMap: removing container" podUID="57c15b3a-53dd-4554-8141-b8a1ec58727e" containerName="revoke-keys"
Oct 30 21:07:16.476043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.476019    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c15b3a-53dd-4554-8141-b8a1ec58727e" containerName="revoke-keys"
Oct 30 21:07:16.476207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.476193    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="354df1b9-f265-40bf-bb6a-0e3bffa8dbb6" containerName="revoke-keys"
Oct 30 21:07:16.476242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.476219    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c15b3a-53dd-4554-8141-b8a1ec58727e" containerName="revoke-keys"
Oct 30 21:07:16.476242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.476230    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a872ae-eb38-47a4-8410-b9928f1d94be" containerName="revoke-keys"
Oct 30 21:07:16.485630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.485291    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:07:16.492461 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.492412    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"]
Oct 30 21:07:16.537692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.537662    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5530ac61-52fa-400c-9506-5a7ea1edb7db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn\" (UID: \"5530ac61-52fa-400c-9506-5a7ea1edb7db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:07:16.537875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.537843    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnzvc\" (UniqueName: \"kubernetes.io/projected/5530ac61-52fa-400c-9506-5a7ea1edb7db-kube-api-access-bnzvc\") pod \"payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn\" (UID: \"5530ac61-52fa-400c-9506-5a7ea1edb7db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:07:16.572935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.569933    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a574c2-dockercfg-7n895\""
Oct 30 21:07:16.572935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.570034    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a574c2\""
Oct 30 21:07:16.677545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.675523    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzvc\" (UniqueName: \"kubernetes.io/projected/5530ac61-52fa-400c-9506-5a7ea1edb7db-kube-api-access-bnzvc\") pod \"payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn\" (UID: \"5530ac61-52fa-400c-9506-5a7ea1edb7db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:07:16.677545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.675747    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5530ac61-52fa-400c-9506-5a7ea1edb7db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn\" (UID: \"5530ac61-52fa-400c-9506-5a7ea1edb7db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:07:16.677545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.676719    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5530ac61-52fa-400c-9506-5a7ea1edb7db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn\" (UID: \"5530ac61-52fa-400c-9506-5a7ea1edb7db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:07:16.729959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.727970    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnzvc\" (UniqueName: \"kubernetes.io/projected/5530ac61-52fa-400c-9506-5a7ea1edb7db-kube-api-access-bnzvc\") pod \"payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn\" (UID: \"5530ac61-52fa-400c-9506-5a7ea1edb7db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:07:16.810660 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:16.810611    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:07:17.101419 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:17.101393    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5530ac61_52fa_400c_9506_5a7ea1edb7db.slice/crio-7c920e7446d4e1a9e0dbb795f6792e785749d0caa9de2d07ad728f52175b4aa2 WatchSource:0}: Error finding container 7c920e7446d4e1a9e0dbb795f6792e785749d0caa9de2d07ad728f52175b4aa2: Status 404 returned error can't find the container with id 7c920e7446d4e1a9e0dbb795f6792e785749d0caa9de2d07ad728f52175b4aa2
Oct 30 21:07:17.101830 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:17.101806    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"]
Oct 30 21:07:17.330456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:17.330419    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn" event={"ID":"5530ac61-52fa-400c-9506-5a7ea1edb7db","Type":"ContainerStarted","Data":"8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604"}
Oct 30 21:07:17.330621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:17.330477    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn" event={"ID":"5530ac61-52fa-400c-9506-5a7ea1edb7db","Type":"ContainerStarted","Data":"7c920e7446d4e1a9e0dbb795f6792e785749d0caa9de2d07ad728f52175b4aa2"}
Oct 30 21:07:17.330621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:17.330504    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:07:17.348564 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:17.348136    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn" podStartSLOduration=1.348120623 podStartE2EDuration="1.348120623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:17.347476021 +0000 UTC m=+1803.839292568" watchObservedRunningTime="2026-10-30 21:07:17.348120623 +0000 UTC m=+1803.839937170"
Oct 30 21:07:19.371372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.371336    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"]
Oct 30 21:07:19.376035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.376010    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.380480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.380443    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:07:19.380979 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.380950    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:07:19.386498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.385956    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"]
Oct 30 21:07:19.533882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.527996    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4810c1-r2sgm\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.533882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.528145    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4810c1-r2sgm\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.533882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.528193    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x92r6\" (UniqueName: \"kubernetes.io/projected/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-kube-api-access-x92r6\") pod \"maas-api-revoke-sub-7e4810c1-r2sgm\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.630188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.630150    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4810c1-r2sgm\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.630355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.630284    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4810c1-r2sgm\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.630355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.630325    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x92r6\" (UniqueName: \"kubernetes.io/projected/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-kube-api-access-x92r6\") pod \"maas-api-revoke-sub-7e4810c1-r2sgm\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.631636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.631609    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4810c1-r2sgm\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.644609 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.644544    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x92r6\" (UniqueName: \"kubernetes.io/projected/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-kube-api-access-x92r6\") pod \"maas-api-revoke-sub-7e4810c1-r2sgm\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.646260 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.646129    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4810c1-r2sgm\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.701159 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.701118    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:19.727043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.727014    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"]
Oct 30 21:07:19.742591 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.741984    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:19.742972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.742946    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"]
Oct 30 21:07:19.834539 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.833974    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c55bbb2e-chk9z\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:19.834539 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.834238    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfb79\" (UniqueName: \"kubernetes.io/projected/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-kube-api-access-wfb79\") pod \"maas-api-revoke-sub-c55bbb2e-chk9z\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:19.834539 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.834297    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c55bbb2e-chk9z\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:19.935424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.935347    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c55bbb2e-chk9z\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:19.935568 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.935525    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfb79\" (UniqueName: \"kubernetes.io/projected/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-kube-api-access-wfb79\") pod \"maas-api-revoke-sub-c55bbb2e-chk9z\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:19.935568 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.935559    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c55bbb2e-chk9z\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:19.936185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.936168    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c55bbb2e-chk9z\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:19.937748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.937724    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c55bbb2e-chk9z\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:19.944656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:19.944636    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfb79\" (UniqueName: \"kubernetes.io/projected/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-kube-api-access-wfb79\") pod \"maas-api-revoke-sub-c55bbb2e-chk9z\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:20.030284 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:20.030256    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff847ad_b0b8_4a60_b23e_6b0c09d273e9.slice/crio-5c8bdba0833e126840c27409bf2e5fa783d1f944f89dd685e77a223dd21c7200 WatchSource:0}: Error finding container 5c8bdba0833e126840c27409bf2e5fa783d1f944f89dd685e77a223dd21c7200: Status 404 returned error can't find the container with id 5c8bdba0833e126840c27409bf2e5fa783d1f944f89dd685e77a223dd21c7200
Oct 30 21:07:20.030369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.030280    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"]
Oct 30 21:07:20.051258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.051232    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"]
Oct 30 21:07:20.057130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.057111    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.065039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.065015    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"]
Oct 30 21:07:20.092244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.092217    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:20.242897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.240055    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003e64de-qvx7l\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.242897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.240164    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69zl\" (UniqueName: \"kubernetes.io/projected/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-kube-api-access-k69zl\") pod \"maas-api-revoke-sub-003e64de-qvx7l\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.242897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.240304    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003e64de-qvx7l\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.341542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.341507    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003e64de-qvx7l\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.341724 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.341599    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69zl\" (UniqueName: \"kubernetes.io/projected/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-kube-api-access-k69zl\") pod \"maas-api-revoke-sub-003e64de-qvx7l\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.341795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.341731    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003e64de-qvx7l\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.342231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.342213    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003e64de-qvx7l\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.344191 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.344173    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003e64de-qvx7l\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.350520 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.350493    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69zl\" (UniqueName: \"kubernetes.io/projected/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-kube-api-access-k69zl\") pod \"maas-api-revoke-sub-003e64de-qvx7l\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.355311 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.355288    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm" event={"ID":"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9","Type":"ContainerStarted","Data":"5c8bdba0833e126840c27409bf2e5fa783d1f944f89dd685e77a223dd21c7200"}
Oct 30 21:07:20.386060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.386034    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:20.448381 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:20.448340    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5871ce_de61_4a4f_bd21_0393ef7f60bd.slice/crio-9fc70ac6032f25590ba89d643b82eebe6b2bbf4a01e52cf09a513d84d56d7293 WatchSource:0}: Error finding container 9fc70ac6032f25590ba89d643b82eebe6b2bbf4a01e52cf09a513d84d56d7293: Status 404 returned error can't find the container with id 9fc70ac6032f25590ba89d643b82eebe6b2bbf4a01e52cf09a513d84d56d7293
Oct 30 21:07:20.449718 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.448925    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"]
Oct 30 21:07:20.679946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:20.679915    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"]
Oct 30 21:07:20.687914 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:20.687563    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8e3c84_34a0_4a16_ac5f_2fa171c35f90.slice/crio-466d4ac649b7063e76e3212b95d3e9173e59809bf224bf6df67ae7e64bde9447 WatchSource:0}: Error finding container 466d4ac649b7063e76e3212b95d3e9173e59809bf224bf6df67ae7e64bde9447: Status 404 returned error can't find the container with id 466d4ac649b7063e76e3212b95d3e9173e59809bf224bf6df67ae7e64bde9447
Oct 30 21:07:21.366820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:21.366732    4810 generic.go:356] "Generic (PLEG): container finished" podID="6ff847ad-b0b8-4a60-b23e-6b0c09d273e9" containerID="dfe4878f7dd9845c5adead19781b537858515a4593cc14dd8f0dd49eb8661cdb" exitCode=0
Oct 30 21:07:21.366990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:21.366821    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm" event={"ID":"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9","Type":"ContainerDied","Data":"dfe4878f7dd9845c5adead19781b537858515a4593cc14dd8f0dd49eb8661cdb"}
Oct 30 21:07:21.368550 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:21.368518    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z" event={"ID":"fb5871ce-de61-4a4f-bd21-0393ef7f60bd","Type":"ContainerStarted","Data":"b63b418e26361cbe56565ec73b8d4566ed3320309667503f99392bb7663d770a"}
Oct 30 21:07:21.368675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:21.368568    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z" event={"ID":"fb5871ce-de61-4a4f-bd21-0393ef7f60bd","Type":"ContainerStarted","Data":"9fc70ac6032f25590ba89d643b82eebe6b2bbf4a01e52cf09a513d84d56d7293"}
Oct 30 21:07:21.370251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:21.370228    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l" event={"ID":"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90","Type":"ContainerStarted","Data":"5849d8e720e935543a2eaa31cb3abbfc1d3ba002c226f9efe19c08ac867c07b8"}
Oct 30 21:07:21.370346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:21.370271    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l" event={"ID":"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90","Type":"ContainerStarted","Data":"466d4ac649b7063e76e3212b95d3e9173e59809bf224bf6df67ae7e64bde9447"}
Oct 30 21:07:21.407914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:21.407870    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z" podStartSLOduration=2.407845423 podStartE2EDuration="2.407845423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:21.403434642 +0000 UTC m=+1807.895251402" watchObservedRunningTime="2026-10-30 21:07:21.407845423 +0000 UTC m=+1807.899661967"
Oct 30 21:07:21.418391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:21.418212    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l" podStartSLOduration=1.418197984 podStartE2EDuration="1.418197984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:21.418085458 +0000 UTC m=+1807.909902002" watchObservedRunningTime="2026-10-30 21:07:21.418197984 +0000 UTC m=+1807.910014530"
Oct 30 21:07:22.381619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.381585    4810 generic.go:356] "Generic (PLEG): container finished" podID="fc8e3c84-34a0-4a16-ac5f-2fa171c35f90" containerID="5849d8e720e935543a2eaa31cb3abbfc1d3ba002c226f9efe19c08ac867c07b8" exitCode=0
Oct 30 21:07:22.381821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.381675    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l" event={"ID":"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90","Type":"ContainerDied","Data":"5849d8e720e935543a2eaa31cb3abbfc1d3ba002c226f9efe19c08ac867c07b8"}
Oct 30 21:07:22.383505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.383479    4810 generic.go:356] "Generic (PLEG): container finished" podID="fb5871ce-de61-4a4f-bd21-0393ef7f60bd" containerID="b63b418e26361cbe56565ec73b8d4566ed3320309667503f99392bb7663d770a" exitCode=0
Oct 30 21:07:22.383634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.383568    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z" event={"ID":"fb5871ce-de61-4a4f-bd21-0393ef7f60bd","Type":"ContainerDied","Data":"b63b418e26361cbe56565ec73b8d4566ed3320309667503f99392bb7663d770a"}
Oct 30 21:07:22.564486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.564459    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:22.574736 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.574709    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-kube-api-access-x92r6\" (UniqueName: \"kubernetes.io/projected/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-kube-api-access-x92r6\") pod \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") "
Oct 30 21:07:22.574887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.574786    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-xks-ca\") pod \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") "
Oct 30 21:07:22.574948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.574924    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-openshift-ca\") pod \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\" (UID: \"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9\") "
Oct 30 21:07:22.575270 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.575250    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-openshift-ca" pod "6ff847ad-b0b8-4a60-b23e-6b0c09d273e9" (UID: "6ff847ad-b0b8-4a60-b23e-6b0c09d273e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:22.575542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.575520    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:22.576853 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.576824    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-kube-api-access-x92r6" pod "6ff847ad-b0b8-4a60-b23e-6b0c09d273e9" (UID: "6ff847ad-b0b8-4a60-b23e-6b0c09d273e9"). InnerVolumeSpecName "kube-api-access-x92r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:22.576965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.576869    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-xks-ca" pod "6ff847ad-b0b8-4a60-b23e-6b0c09d273e9" (UID: "6ff847ad-b0b8-4a60-b23e-6b0c09d273e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:22.676380 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.676313    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x92r6\" (UniqueName: \"kubernetes.io/projected/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-kube-api-access-x92r6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:22.676380 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:22.676336    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:23.402491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.402460    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"
Oct 30 21:07:23.402642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.402489    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm" event={"ID":"6ff847ad-b0b8-4a60-b23e-6b0c09d273e9","Type":"ContainerDied","Data":"5c8bdba0833e126840c27409bf2e5fa783d1f944f89dd685e77a223dd21c7200"}
Oct 30 21:07:23.402642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.402556    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c8bdba0833e126840c27409bf2e5fa783d1f944f89dd685e77a223dd21c7200"
Oct 30 21:07:23.575980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.575954    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:23.614326 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.613598    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-xks-ca\") pod \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") "
Oct 30 21:07:23.614513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.614404    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-openshift-ca\") pod \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") "
Oct 30 21:07:23.617740 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.615075    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-openshift-ca" pod "fb5871ce-de61-4a4f-bd21-0393ef7f60bd" (UID: "fb5871ce-de61-4a4f-bd21-0393ef7f60bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:23.617740 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.615244    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-kube-api-access-wfb79\" (UniqueName: \"kubernetes.io/projected/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-kube-api-access-wfb79\") pod \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\" (UID: \"fb5871ce-de61-4a4f-bd21-0393ef7f60bd\") "
Oct 30 21:07:23.619526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.617796    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:23.622965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.622920    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-xks-ca" pod "fb5871ce-de61-4a4f-bd21-0393ef7f60bd" (UID: "fb5871ce-de61-4a4f-bd21-0393ef7f60bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:23.632846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.625229    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-kube-api-access-wfb79" pod "fb5871ce-de61-4a4f-bd21-0393ef7f60bd" (UID: "fb5871ce-de61-4a4f-bd21-0393ef7f60bd"). InnerVolumeSpecName "kube-api-access-wfb79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:23.688291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.688227    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:23.718668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.718636    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-xks-ca\") pod \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") "
Oct 30 21:07:23.718960 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.718935    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-kube-api-access-k69zl\" (UniqueName: \"kubernetes.io/projected/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-kube-api-access-k69zl\") pod \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") "
Oct 30 21:07:23.719107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.718990    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-openshift-ca\") pod \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\" (UID: \"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90\") "
Oct 30 21:07:23.719484 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.719455    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-openshift-ca" pod "fc8e3c84-34a0-4a16-ac5f-2fa171c35f90" (UID: "fc8e3c84-34a0-4a16-ac5f-2fa171c35f90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:23.719756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.719723    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:23.719880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.719760    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:23.719880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.719795    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfb79\" (UniqueName: \"kubernetes.io/projected/fb5871ce-de61-4a4f-bd21-0393ef7f60bd-kube-api-access-wfb79\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:23.721799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.721779    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-kube-api-access-k69zl" pod "fc8e3c84-34a0-4a16-ac5f-2fa171c35f90" (UID: "fc8e3c84-34a0-4a16-ac5f-2fa171c35f90"). InnerVolumeSpecName "kube-api-access-k69zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:23.721968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.721943    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-xks-ca" pod "fc8e3c84-34a0-4a16-ac5f-2fa171c35f90" (UID: "fc8e3c84-34a0-4a16-ac5f-2fa171c35f90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:23.821670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.821631    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69zl\" (UniqueName: \"kubernetes.io/projected/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-kube-api-access-k69zl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:23.821843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:23.821675    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:24.415623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:24.415594    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"
Oct 30 21:07:24.415834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:24.415630    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z" event={"ID":"fb5871ce-de61-4a4f-bd21-0393ef7f60bd","Type":"ContainerDied","Data":"9fc70ac6032f25590ba89d643b82eebe6b2bbf4a01e52cf09a513d84d56d7293"}
Oct 30 21:07:24.415834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:24.415679    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fc70ac6032f25590ba89d643b82eebe6b2bbf4a01e52cf09a513d84d56d7293"
Oct 30 21:07:24.417388 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:24.417366    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"
Oct 30 21:07:24.417518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:24.417403    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l" event={"ID":"fc8e3c84-34a0-4a16-ac5f-2fa171c35f90","Type":"ContainerDied","Data":"466d4ac649b7063e76e3212b95d3e9173e59809bf224bf6df67ae7e64bde9447"}
Oct 30 21:07:24.417518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:24.417458    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="466d4ac649b7063e76e3212b95d3e9173e59809bf224bf6df67ae7e64bde9447"
Oct 30 21:07:25.158542 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:25.158505    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:07:28.346148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:28.346118    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:07:31.774170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:31.774131    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"]
Oct 30 21:07:32.190906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:32.186976    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"]
Oct 30 21:07:35.120880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:35.119318    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"]
Oct 30 21:07:38.531619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:38.530848    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"]
Oct 30 21:07:38.675607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:38.675563    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:07:41.110517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.110478    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"]
Oct 30 21:07:41.114227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.114202    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-89d993-6c2x2"]
Oct 30 21:07:41.665293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.665261    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a872ae-eb38-47a4-8410-b9928f1d94be" path="/var/lib/kubelet/pods/a5a872ae-eb38-47a4-8410-b9928f1d94be/volumes"
Oct 30 21:07:41.890205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.890168    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-0f8dd3"]
Oct 30 21:07:41.891195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.891173    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6ff847ad-b0b8-4a60-b23e-6b0c09d273e9" containerName="revoke-keys"
Oct 30 21:07:41.891195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.891193    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff847ad-b0b8-4a60-b23e-6b0c09d273e9" containerName="revoke-keys"
Oct 30 21:07:41.891293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.891252    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fb5871ce-de61-4a4f-bd21-0393ef7f60bd" containerName="revoke-keys"
Oct 30 21:07:41.891293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.891260    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5871ce-de61-4a4f-bd21-0393ef7f60bd" containerName="revoke-keys"
Oct 30 21:07:41.891293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.891284    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fc8e3c84-34a0-4a16-ac5f-2fa171c35f90" containerName="revoke-keys"
Oct 30 21:07:41.891293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.891290    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8e3c84-34a0-4a16-ac5f-2fa171c35f90" containerName="revoke-keys"
Oct 30 21:07:41.891435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.891424    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff847ad-b0b8-4a60-b23e-6b0c09d273e9" containerName="revoke-keys"
Oct 30 21:07:41.891470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.891450    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5871ce-de61-4a4f-bd21-0393ef7f60bd" containerName="revoke-keys"
Oct 30 21:07:41.891470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.891462    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8e3c84-34a0-4a16-ac5f-2fa171c35f90" containerName="revoke-keys"
Oct 30 21:07:41.904433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.903547    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-0f8dd3"]
Oct 30 21:07:41.904433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.903650    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-0f8dd3"
Oct 30 21:07:41.913369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.913326    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrzx8\""
Oct 30 21:07:41.942910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:41.942845    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmvg\" (UniqueName: \"kubernetes.io/projected/191ad20d-4e76-4273-9b2b-8cc09774aad5-kube-api-access-lnmvg\") pod \"test-curl-7603-0f8dd3\" (UID: \"191ad20d-4e76-4273-9b2b-8cc09774aad5\") " pod="opendatahub/test-curl-7603-0f8dd3"
Oct 30 21:07:42.044535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:42.044502    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmvg\" (UniqueName: \"kubernetes.io/projected/191ad20d-4e76-4273-9b2b-8cc09774aad5-kube-api-access-lnmvg\") pod \"test-curl-7603-0f8dd3\" (UID: \"191ad20d-4e76-4273-9b2b-8cc09774aad5\") " pod="opendatahub/test-curl-7603-0f8dd3"
Oct 30 21:07:42.054549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:42.054512    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmvg\" (UniqueName: \"kubernetes.io/projected/191ad20d-4e76-4273-9b2b-8cc09774aad5-kube-api-access-lnmvg\") pod \"test-curl-7603-0f8dd3\" (UID: \"191ad20d-4e76-4273-9b2b-8cc09774aad5\") " pod="opendatahub/test-curl-7603-0f8dd3"
Oct 30 21:07:42.224582 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:42.224495    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-0f8dd3"
Oct 30 21:07:42.528455 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:42.528423    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-0f8dd3"]
Oct 30 21:07:42.532529 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:42.532365    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod191ad20d_4e76_4273_9b2b_8cc09774aad5.slice/crio-e241fb4b93a05bcb29b9279811bf3cf1abfef9ded87a38a0a1546451b99ec13f WatchSource:0}: Error finding container e241fb4b93a05bcb29b9279811bf3cf1abfef9ded87a38a0a1546451b99ec13f: Status 404 returned error can't find the container with id e241fb4b93a05bcb29b9279811bf3cf1abfef9ded87a38a0a1546451b99ec13f
Oct 30 21:07:42.637055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:42.637020    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-0f8dd3" event={"ID":"191ad20d-4e76-4273-9b2b-8cc09774aad5","Type":"ContainerStarted","Data":"31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed"}
Oct 30 21:07:42.637196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:42.637071    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-0f8dd3" event={"ID":"191ad20d-4e76-4273-9b2b-8cc09774aad5","Type":"ContainerStarted","Data":"e241fb4b93a05bcb29b9279811bf3cf1abfef9ded87a38a0a1546451b99ec13f"}
Oct 30 21:07:42.654209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:42.654168    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7603-0f8dd3" podStartSLOduration=1.654151008 podStartE2EDuration="1.654151008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:42.652398136 +0000 UTC m=+1829.144214681" watchObservedRunningTime="2026-10-30 21:07:42.654151008 +0000 UTC m=+1829.145967551"
Oct 30 21:07:43.326409 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.326377    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-0f8dd3"]
Oct 30 21:07:43.338886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.333621    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-0f8dd3"]
Oct 30 21:07:43.526871 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.526827    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"]
Oct 30 21:07:43.527990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.527963    4810 cpu_manager.go:414] "containerMap: removing container" podUID="191ad20d-4e76-4273-9b2b-8cc09774aad5" containerName="test-curl-7603-0f8dd3"
Oct 30 21:07:43.527990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.527988    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191ad20d-4e76-4273-9b2b-8cc09774aad5" containerName="test-curl-7603-0f8dd3"
Oct 30 21:07:43.528206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.528173    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="191ad20d-4e76-4273-9b2b-8cc09774aad5" containerName="test-curl-7603-0f8dd3"
Oct 30 21:07:43.532847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.532821    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:43.539579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.538055    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:07:43.539579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.538143    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:07:43.555330 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.553641    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"]
Oct 30 21:07:43.595036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.594980    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f91ecb99-mvftp\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:43.595228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.595211    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f91ecb99-mvftp\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:43.595370 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.595313    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwhn\" (UniqueName: \"kubernetes.io/projected/b55e2fea-36c7-4d70-8962-bfc58ea292ed-kube-api-access-wkwhn\") pod \"maas-api-revoke-sub-f91ecb99-mvftp\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:43.696331 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.696301    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f91ecb99-mvftp\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:43.696595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.696350    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwhn\" (UniqueName: \"kubernetes.io/projected/b55e2fea-36c7-4d70-8962-bfc58ea292ed-kube-api-access-wkwhn\") pod \"maas-api-revoke-sub-f91ecb99-mvftp\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:43.696595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.696431    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f91ecb99-mvftp\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:43.697128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.697102    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f91ecb99-mvftp\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:43.698925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.698903    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f91ecb99-mvftp\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:43.705387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.705362    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwhn\" (UniqueName: \"kubernetes.io/projected/b55e2fea-36c7-4d70-8962-bfc58ea292ed-kube-api-access-wkwhn\") pod \"maas-api-revoke-sub-f91ecb99-mvftp\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:43.854239 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:43.854170    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:44.156907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:44.153056    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"]
Oct 30 21:07:44.156907 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:44.154384    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55e2fea_36c7_4d70_8962_bfc58ea292ed.slice/crio-e297239d83331ce3662814632de5c36211f944e5740a606f922e483ceb6a4901 WatchSource:0}: Error finding container e297239d83331ce3662814632de5c36211f944e5740a606f922e483ceb6a4901: Status 404 returned error can't find the container with id e297239d83331ce3662814632de5c36211f944e5740a606f922e483ceb6a4901
Oct 30 21:07:44.657683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:44.657643    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp" event={"ID":"b55e2fea-36c7-4d70-8962-bfc58ea292ed","Type":"ContainerStarted","Data":"6e5eb7d9ee1c53297d0c2e90c5fd26df681775d9360eecaa02d2e3bc7890c3d7"}
Oct 30 21:07:44.658135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:44.657737    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp" event={"ID":"b55e2fea-36c7-4d70-8962-bfc58ea292ed","Type":"ContainerStarted","Data":"e297239d83331ce3662814632de5c36211f944e5740a606f922e483ceb6a4901"}
Oct 30 21:07:44.658135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:44.657660    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7603-0f8dd3" podUID="191ad20d-4e76-4273-9b2b-8cc09774aad5" containerName="test-curl-7603-0f8dd3" containerID="cri-o://31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed" gracePeriod=2
Oct 30 21:07:44.682473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:44.682426    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp" podStartSLOduration=1.682414026 podStartE2EDuration="1.682414026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:44.679475065 +0000 UTC m=+1831.171291609" watchObservedRunningTime="2026-10-30 21:07:44.682414026 +0000 UTC m=+1831.174230569"
Oct 30 21:07:45.669459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.669424    4810 generic.go:356] "Generic (PLEG): container finished" podID="b55e2fea-36c7-4d70-8962-bfc58ea292ed" containerID="6e5eb7d9ee1c53297d0c2e90c5fd26df681775d9360eecaa02d2e3bc7890c3d7" exitCode=0
Oct 30 21:07:45.669976 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.669528    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp" event={"ID":"b55e2fea-36c7-4d70-8962-bfc58ea292ed","Type":"ContainerDied","Data":"6e5eb7d9ee1c53297d0c2e90c5fd26df681775d9360eecaa02d2e3bc7890c3d7"}
Oct 30 21:07:45.757295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.757264    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"]
Oct 30 21:07:45.761968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.761944    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:07:45.774897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.774126    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-99668e\""
Oct 30 21:07:45.779116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.779093    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"]
Oct 30 21:07:45.793873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.793837    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"]
Oct 30 21:07:45.798779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.798754    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:07:45.816247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.813688    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"]
Oct 30 21:07:45.825566 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.825535    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx4f8\" (UniqueName: \"kubernetes.io/projected/e6089936-7fbd-42fa-b030-20e1af0a8e2b-kube-api-access-zx4f8\") pod \"payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks\" (UID: \"e6089936-7fbd-42fa-b030-20e1af0a8e2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:07:45.825683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.825578    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb1e381-15d1-4be5-ae3b-995385a56b10-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt\" (UID: \"3fb1e381-15d1-4be5-ae3b-995385a56b10\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:07:45.825683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.825611    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gg64\" (UniqueName: \"kubernetes.io/projected/3fb1e381-15d1-4be5-ae3b-995385a56b10-kube-api-access-2gg64\") pod \"payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt\" (UID: \"3fb1e381-15d1-4be5-ae3b-995385a56b10\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:07:45.825802 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.825785    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6089936-7fbd-42fa-b030-20e1af0a8e2b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks\" (UID: \"e6089936-7fbd-42fa-b030-20e1af0a8e2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:07:45.843378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.843350    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-99668e-dockercfg-s9xw4\""
Oct 30 21:07:45.948926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.948186    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx4f8\" (UniqueName: \"kubernetes.io/projected/e6089936-7fbd-42fa-b030-20e1af0a8e2b-kube-api-access-zx4f8\") pod \"payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks\" (UID: \"e6089936-7fbd-42fa-b030-20e1af0a8e2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:07:45.948926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.948254    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb1e381-15d1-4be5-ae3b-995385a56b10-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt\" (UID: \"3fb1e381-15d1-4be5-ae3b-995385a56b10\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:07:45.948926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.948285    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gg64\" (UniqueName: \"kubernetes.io/projected/3fb1e381-15d1-4be5-ae3b-995385a56b10-kube-api-access-2gg64\") pod \"payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt\" (UID: \"3fb1e381-15d1-4be5-ae3b-995385a56b10\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:07:45.948926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.948475    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6089936-7fbd-42fa-b030-20e1af0a8e2b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks\" (UID: \"e6089936-7fbd-42fa-b030-20e1af0a8e2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:07:45.950414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.949476    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6089936-7fbd-42fa-b030-20e1af0a8e2b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks\" (UID: \"e6089936-7fbd-42fa-b030-20e1af0a8e2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:07:45.950629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.950597    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb1e381-15d1-4be5-ae3b-995385a56b10-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt\" (UID: \"3fb1e381-15d1-4be5-ae3b-995385a56b10\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:07:45.971616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.971576    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gg64\" (UniqueName: \"kubernetes.io/projected/3fb1e381-15d1-4be5-ae3b-995385a56b10-kube-api-access-2gg64\") pod \"payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt\" (UID: \"3fb1e381-15d1-4be5-ae3b-995385a56b10\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:07:45.992054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:45.982663    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx4f8\" (UniqueName: \"kubernetes.io/projected/e6089936-7fbd-42fa-b030-20e1af0a8e2b-kube-api-access-zx4f8\") pod \"payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks\" (UID: \"e6089936-7fbd-42fa-b030-20e1af0a8e2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:07:46.083561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.083167    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:07:46.120919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.120888    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:07:46.405998 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:46.405963    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6089936_7fbd_42fa_b030_20e1af0a8e2b.slice/crio-33fb9ebac34cdfc377c262a939e1845d69d3d01df05e498b415340b152f3ccc5 WatchSource:0}: Error finding container 33fb9ebac34cdfc377c262a939e1845d69d3d01df05e498b415340b152f3ccc5: Status 404 returned error can't find the container with id 33fb9ebac34cdfc377c262a939e1845d69d3d01df05e498b415340b152f3ccc5
Oct 30 21:07:46.407751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.407713    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"]
Oct 30 21:07:46.480653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.480591    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"]
Oct 30 21:07:46.563009 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.562978    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"]
Oct 30 21:07:46.686533 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.686493    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt" event={"ID":"3fb1e381-15d1-4be5-ae3b-995385a56b10","Type":"ContainerStarted","Data":"1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407"}
Oct 30 21:07:46.687023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.686561    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt" event={"ID":"3fb1e381-15d1-4be5-ae3b-995385a56b10","Type":"ContainerStarted","Data":"31e2e57c8fbcf7c31c04bfa3474779580c7d7988c18bfc0e46f2a967d4e678aa"}
Oct 30 21:07:46.687023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.686589    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:07:46.688641 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.688600    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks" event={"ID":"e6089936-7fbd-42fa-b030-20e1af0a8e2b","Type":"ContainerStarted","Data":"00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789"}
Oct 30 21:07:46.688843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.688665    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks" event={"ID":"e6089936-7fbd-42fa-b030-20e1af0a8e2b","Type":"ContainerStarted","Data":"33fb9ebac34cdfc377c262a939e1845d69d3d01df05e498b415340b152f3ccc5"}
Oct 30 21:07:46.688843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.688697    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:07:46.708602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.708551    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt" podStartSLOduration=1.708534435 podStartE2EDuration="1.708534435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:46.703255084 +0000 UTC m=+1833.195071628" watchObservedRunningTime="2026-10-30 21:07:46.708534435 +0000 UTC m=+1833.200350977"
Oct 30 21:07:46.730752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.730672    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks" podStartSLOduration=1.7306582019999999 podStartE2EDuration="1.730658202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:46.724971444 +0000 UTC m=+1833.216788003" watchObservedRunningTime="2026-10-30 21:07:46.730658202 +0000 UTC m=+1833.222474745"
Oct 30 21:07:46.833082 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.833046    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg" podUID="8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" containerName="main" containerID="cri-o://f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8" gracePeriod=45
Oct 30 21:07:46.881052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.880789    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg" podUID="8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" containerName="main" probeResult="failure" output="Get \"https://10.132.0.197:8000/ready\": dial tcp 10.132.0.197:8000: connect: connection refused"
Oct 30 21:07:46.950600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.950287    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:46.955641 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:46.955611    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-0f8dd3"
Oct 30 21:07:47.085364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.085325    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191ad20d-4e76-4273-9b2b-8cc09774aad5-kube-api-access-lnmvg\" (UniqueName: \"kubernetes.io/projected/191ad20d-4e76-4273-9b2b-8cc09774aad5-kube-api-access-lnmvg\") pod \"191ad20d-4e76-4273-9b2b-8cc09774aad5\" (UID: \"191ad20d-4e76-4273-9b2b-8cc09774aad5\") "
Oct 30 21:07:47.085518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.085500    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b55e2fea-36c7-4d70-8962-bfc58ea292ed-kube-api-access-wkwhn\" (UniqueName: \"kubernetes.io/projected/b55e2fea-36c7-4d70-8962-bfc58ea292ed-kube-api-access-wkwhn\") pod \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") "
Oct 30 21:07:47.085594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.085579    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-openshift-ca\") pod \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") "
Oct 30 21:07:47.085657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.085615    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-xks-ca\") pod \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\" (UID: \"b55e2fea-36c7-4d70-8962-bfc58ea292ed\") "
Oct 30 21:07:47.094981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.094947    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-xks-ca" pod "b55e2fea-36c7-4d70-8962-bfc58ea292ed" (UID: "b55e2fea-36c7-4d70-8962-bfc58ea292ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:47.095429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.095403    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-openshift-ca" pod "b55e2fea-36c7-4d70-8962-bfc58ea292ed" (UID: "b55e2fea-36c7-4d70-8962-bfc58ea292ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:47.095905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.095832    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191ad20d-4e76-4273-9b2b-8cc09774aad5-kube-api-access-lnmvg" pod "191ad20d-4e76-4273-9b2b-8cc09774aad5" (UID: "191ad20d-4e76-4273-9b2b-8cc09774aad5"). InnerVolumeSpecName "kube-api-access-lnmvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:47.096154 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.096121    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55e2fea-36c7-4d70-8962-bfc58ea292ed-kube-api-access-wkwhn" pod "b55e2fea-36c7-4d70-8962-bfc58ea292ed" (UID: "b55e2fea-36c7-4d70-8962-bfc58ea292ed"). InnerVolumeSpecName "kube-api-access-wkwhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:47.184705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.184675    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:07:47.187110 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.187084    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkwhn\" (UniqueName: \"kubernetes.io/projected/b55e2fea-36c7-4d70-8962-bfc58ea292ed-kube-api-access-wkwhn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.187274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.187259    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.187369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.187357    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b55e2fea-36c7-4d70-8962-bfc58ea292ed-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.187459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.187448    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmvg\" (UniqueName: \"kubernetes.io/projected/191ad20d-4e76-4273-9b2b-8cc09774aad5-kube-api-access-lnmvg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.229292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.229233    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64" podUID="c39858eb-242d-4afe-979a-74216af0dce4" containerName="main" containerID="cri-o://aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169" gracePeriod=45
Oct 30 21:07:47.288617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.288589    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-dshm\") pod \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") "
Oct 30 21:07:47.288815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.288631    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-model-cache\") pod \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") "
Oct 30 21:07:47.288815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.288671    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tmp-dir\") pod \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") "
Oct 30 21:07:47.288987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.288935    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-kube-api-access-dsgd7\" (UniqueName: \"kubernetes.io/projected/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-kube-api-access-dsgd7\") pod \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") "
Oct 30 21:07:47.288987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.288951    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-model-cache" pod "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" (UID: "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:47.289110 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.289090    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tmp-dir" pod "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" (UID: "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:47.289264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.289246    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-home\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-home\") pod \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") "
Oct 30 21:07:47.289448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.289432    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tls-certs\") pod \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\" (UID: \"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709\") "
Oct 30 21:07:47.289610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.289461    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-home" pod "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" (UID: "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:47.290337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.290311    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.290433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.290342    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.290433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.290379    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.291343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.291313    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-kube-api-access-dsgd7" pod "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" (UID: "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709"). InnerVolumeSpecName "kube-api-access-dsgd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:47.291443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.291379    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-dshm" pod "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" (UID: "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:47.291659 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.291639    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tls-certs" pod "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" (UID: "8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:47.391706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.391671    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsgd7\" (UniqueName: \"kubernetes.io/projected/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-kube-api-access-dsgd7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.391706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.391707    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.391978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.391719    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.511020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.510989    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:07:47.595040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.595011    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-dshm\") pod \"c39858eb-242d-4afe-979a-74216af0dce4\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") "
Oct 30 21:07:47.595219 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.595192    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-model-cache\") pod \"c39858eb-242d-4afe-979a-74216af0dce4\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") "
Oct 30 21:07:47.595327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.595312    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c39858eb-242d-4afe-979a-74216af0dce4-tls-certs\" (UniqueName: \"kubernetes.io/secret/c39858eb-242d-4afe-979a-74216af0dce4-tls-certs\") pod \"c39858eb-242d-4afe-979a-74216af0dce4\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") "
Oct 30 21:07:47.595402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.595374    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39858eb-242d-4afe-979a-74216af0dce4-kube-api-access-fhhgv\" (UniqueName: \"kubernetes.io/projected/c39858eb-242d-4afe-979a-74216af0dce4-kube-api-access-fhhgv\") pod \"c39858eb-242d-4afe-979a-74216af0dce4\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") "
Oct 30 21:07:47.595454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.595414    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-home\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-home\") pod \"c39858eb-242d-4afe-979a-74216af0dce4\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") "
Oct 30 21:07:47.595454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.595424    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-model-cache" pod "c39858eb-242d-4afe-979a-74216af0dce4" (UID: "c39858eb-242d-4afe-979a-74216af0dce4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:47.595559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.595459    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-tmp-dir\") pod \"c39858eb-242d-4afe-979a-74216af0dce4\" (UID: \"c39858eb-242d-4afe-979a-74216af0dce4\") "
Oct 30 21:07:47.595949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.595891    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-home" pod "c39858eb-242d-4afe-979a-74216af0dce4" (UID: "c39858eb-242d-4afe-979a-74216af0dce4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:47.595949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.595925    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-tmp-dir" pod "c39858eb-242d-4afe-979a-74216af0dce4" (UID: "c39858eb-242d-4afe-979a-74216af0dce4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:47.596143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.596127    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.596197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.596146    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.596197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.596156    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.599142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.599116    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-dshm" pod "c39858eb-242d-4afe-979a-74216af0dce4" (UID: "c39858eb-242d-4afe-979a-74216af0dce4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:47.599142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.599136    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39858eb-242d-4afe-979a-74216af0dce4-kube-api-access-fhhgv" pod "c39858eb-242d-4afe-979a-74216af0dce4" (UID: "c39858eb-242d-4afe-979a-74216af0dce4"). InnerVolumeSpecName "kube-api-access-fhhgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:47.600700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.600679    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39858eb-242d-4afe-979a-74216af0dce4-tls-certs" pod "c39858eb-242d-4afe-979a-74216af0dce4" (UID: "c39858eb-242d-4afe-979a-74216af0dce4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:47.663870 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.663832    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191ad20d-4e76-4273-9b2b-8cc09774aad5" path="/var/lib/kubelet/pods/191ad20d-4e76-4273-9b2b-8cc09774aad5/volumes"
Oct 30 21:07:47.698682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.698560    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c39858eb-242d-4afe-979a-74216af0dce4-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.698682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.698595    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39858eb-242d-4afe-979a-74216af0dce4-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.698682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.698612    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhhgv\" (UniqueName: \"kubernetes.io/projected/c39858eb-242d-4afe-979a-74216af0dce4-kube-api-access-fhhgv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:47.727693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.727652    4810 generic.go:356] "Generic (PLEG): container finished" podID="191ad20d-4e76-4273-9b2b-8cc09774aad5" containerID="31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed" exitCode=137
Oct 30 21:07:47.727929 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.727852    4810 scope.go:122] "RemoveContainer" containerID="31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed"
Oct 30 21:07:47.728055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.728039    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-0f8dd3"
Oct 30 21:07:47.744418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.744389    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp" event={"ID":"b55e2fea-36c7-4d70-8962-bfc58ea292ed","Type":"ContainerDied","Data":"e297239d83331ce3662814632de5c36211f944e5740a606f922e483ceb6a4901"}
Oct 30 21:07:47.744621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.744607    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e297239d83331ce3662814632de5c36211f944e5740a606f922e483ceb6a4901"
Oct 30 21:07:47.744764 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.744753    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"
Oct 30 21:07:47.750123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.750090    4810 generic.go:356] "Generic (PLEG): container finished" podID="8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" containerID="f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8" exitCode=0
Oct 30 21:07:47.750238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.750224    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg" event={"ID":"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709","Type":"ContainerDied","Data":"f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8"}
Oct 30 21:07:47.750316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.750251    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg" event={"ID":"8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709","Type":"ContainerDied","Data":"72ce24f7ce68d2493a561955280ead04e2e24e53c4169a9df0ece7dd7782e182"}
Oct 30 21:07:47.750316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.750301    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"
Oct 30 21:07:47.753480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.753410    4810 generic.go:356] "Generic (PLEG): container finished" podID="c39858eb-242d-4afe-979a-74216af0dce4" containerID="aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169" exitCode=0
Oct 30 21:07:47.753480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.753450    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"
Oct 30 21:07:47.753644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.753491    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64" event={"ID":"c39858eb-242d-4afe-979a-74216af0dce4","Type":"ContainerDied","Data":"aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169"}
Oct 30 21:07:47.753644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.753527    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64" event={"ID":"c39858eb-242d-4afe-979a-74216af0dce4","Type":"ContainerDied","Data":"ebe2c8c6d9a2ae8c3a466d65681cbf67ae1debd229c153404757be11fee7b8bd"}
Oct 30 21:07:47.757409 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.757393    4810 scope.go:122] "RemoveContainer" containerID="31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed"
Oct 30 21:07:47.757732 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:47.757697    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed\": container with ID starting with 31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed not found: ID does not exist" containerID="31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed"
Oct 30 21:07:47.757821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.757730    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed"} err="failed to get container status \"31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed\": rpc error: code = NotFound desc = could not find container \"31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed\": container with ID starting with 31b1267fe06dfe5940501d539ee41be44fb798ff24281fd48eac18b45ff8ceed not found: ID does not exist"
Oct 30 21:07:47.757821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.757747    4810 scope.go:122] "RemoveContainer" containerID="f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8"
Oct 30 21:07:47.768913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.768892    4810 scope.go:122] "RemoveContainer" containerID="f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8"
Oct 30 21:07:47.769187 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:47.769169    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8\": container with ID starting with f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8 not found: ID does not exist" containerID="f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8"
Oct 30 21:07:47.769250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.769192    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8"} err="failed to get container status \"f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8\": rpc error: code = NotFound desc = could not find container \"f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8\": container with ID starting with f4cd6666e5a0c15e54960081996efc84ab2bd4ad6f6582051e863859fe99a3e8 not found: ID does not exist"
Oct 30 21:07:47.769250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.769208    4810 scope.go:122] "RemoveContainer" containerID="aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169"
Oct 30 21:07:47.778805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.778775    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"]
Oct 30 21:07:47.782829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.782811    4810 scope.go:122] "RemoveContainer" containerID="aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169"
Oct 30 21:07:47.783327 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:47.783100    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169\": container with ID starting with aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169 not found: ID does not exist" containerID="aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169"
Oct 30 21:07:47.783327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.783128    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169"} err="failed to get container status \"aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169\": rpc error: code = NotFound desc = could not find container \"aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169\": container with ID starting with aa4b3c4e149bb370099f712b5eb70527cf9fa0887ccdfcc947b583c360b49169 not found: ID does not exist"
Oct 30 21:07:47.784712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.784688    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8fa15b/auth-test-model-8fa15b-kserve-856f6f8565-ctcwg"]
Oct 30 21:07:47.795596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.795571    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"]
Oct 30 21:07:47.800207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:47.800186    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e53c37/auth-test-model-e53c37-kserve-7897c66b9-mkc64"]
Oct 30 21:07:48.029711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.029624    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"]
Oct 30 21:07:48.033583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.031062    4810 cpu_manager.go:414] "containerMap: removing container" podUID="b55e2fea-36c7-4d70-8962-bfc58ea292ed" containerName="revoke-keys"
Oct 30 21:07:48.033583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.031094    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b55e2fea-36c7-4d70-8962-bfc58ea292ed" containerName="revoke-keys"
Oct 30 21:07:48.033583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.031121    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c39858eb-242d-4afe-979a-74216af0dce4" containerName="main"
Oct 30 21:07:48.033583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.031131    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39858eb-242d-4afe-979a-74216af0dce4" containerName="main"
Oct 30 21:07:48.033583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.031176    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" containerName="main"
Oct 30 21:07:48.033583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.031183    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" containerName="main"
Oct 30 21:07:48.033583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.031365    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" containerName="main"
Oct 30 21:07:48.033583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.031388    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="b55e2fea-36c7-4d70-8962-bfc58ea292ed" containerName="revoke-keys"
Oct 30 21:07:48.033583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.031398    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39858eb-242d-4afe-979a-74216af0dce4" containerName="main"
Oct 30 21:07:48.037213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.037190    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.041794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.041771    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6538cc\"/\"openshift-service-ca.crt\""
Oct 30 21:07:48.042035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.041934    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6538cc\"/\"kube-root-ca.crt\""
Oct 30 21:07:48.042035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.041991    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6538cc\"/\"default-dockercfg-c87kr\""
Oct 30 21:07:48.042215 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.042195    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6538cc\"/\"noten-model-6538cc-kserve-self-signed-certs\""
Oct 30 21:07:48.046487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.046463    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"]
Oct 30 21:07:48.104548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.104510    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.105023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.104775    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-model-cache\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.105023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.104930    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-home\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.105023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.104974    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-tmp-dir\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.105225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.105196    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdmtp\" (UniqueName: \"kubernetes.io/projected/0b222375-55c0-45ad-807a-4e28d4c4399a-kube-api-access-pdmtp\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.105299 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.105251    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-dshm\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.206592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.206540    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdmtp\" (UniqueName: \"kubernetes.io/projected/0b222375-55c0-45ad-807a-4e28d4c4399a-kube-api-access-pdmtp\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.206803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.206600    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-dshm\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.206803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.206746    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.206803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.206770    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-model-cache\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.207015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.206840    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-home\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.207015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.206894    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-tmp-dir\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.207349 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.207291    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-model-cache\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.207693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.207673    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-home\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.207847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.207701    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-tmp-dir\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.209654 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.209631    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-dshm\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.210324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.210299    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.216209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.216183    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdmtp\" (UniqueName: \"kubernetes.io/projected/0b222375-55c0-45ad-807a-4e28d4c4399a-kube-api-access-pdmtp\") pod \"noten-model-6538cc-kserve-6c9c4c44fc-zdhh6\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") " pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.371951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.371827    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:48.759719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:48.759696    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"]
Oct 30 21:07:48.761257 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:48.761218    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b222375_55c0_45ad_807a_4e28d4c4399a.slice/crio-6809a908fcb3148e688dfce4df51cd5e1fc15c41ae4a260c5ee95677e005a8ee WatchSource:0}: Error finding container 6809a908fcb3148e688dfce4df51cd5e1fc15c41ae4a260c5ee95677e005a8ee: Status 404 returned error can't find the container with id 6809a908fcb3148e688dfce4df51cd5e1fc15c41ae4a260c5ee95677e005a8ee
Oct 30 21:07:49.663709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:49.663673    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709" path="/var/lib/kubelet/pods/8fbe7c00-2747-4ca6-ba6c-52a5f9a9a709/volumes"
Oct 30 21:07:49.664362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:49.664344    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39858eb-242d-4afe-979a-74216af0dce4" path="/var/lib/kubelet/pods/c39858eb-242d-4afe-979a-74216af0dce4/volumes"
Oct 30 21:07:49.778696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:49.778660    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" event={"ID":"0b222375-55c0-45ad-807a-4e28d4c4399a","Type":"ContainerStarted","Data":"f34511a029bcf87df8b58c221dbb3690aefb9d91e17b8a48248306d77cf78e8d"}
Oct 30 21:07:49.778696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:49.778707    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" event={"ID":"0b222375-55c0-45ad-807a-4e28d4c4399a","Type":"ContainerStarted","Data":"6809a908fcb3148e688dfce4df51cd5e1fc15c41ae4a260c5ee95677e005a8ee"}
Oct 30 21:07:49.799334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:49.799284    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" podStartSLOduration=1.799270354 podStartE2EDuration="1.799270354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:49.796182251 +0000 UTC m=+1836.287998797" watchObservedRunningTime="2026-10-30 21:07:49.799270354 +0000 UTC m=+1836.291086897"
Oct 30 21:07:50.160398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.160355    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj" podUID="6393d03c-e423-4037-9aef-88430e0674d1" containerName="main" containerID="cri-o://a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653" gracePeriod=45
Oct 30 21:07:50.322398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.322361    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj" podUID="6393d03c-e423-4037-9aef-88430e0674d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": dial tcp 10.132.0.201:8000: connect: connection refused"
Oct 30 21:07:50.374676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.374638    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:07:50.461337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.461312    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:07:50.552751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.552722    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6393d03c-e423-4037-9aef-88430e0674d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/6393d03c-e423-4037-9aef-88430e0674d1-tls-certs\") pod \"6393d03c-e423-4037-9aef-88430e0674d1\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") "
Oct 30 21:07:50.552932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.552766    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-home\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-home\") pod \"6393d03c-e423-4037-9aef-88430e0674d1\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") "
Oct 30 21:07:50.552978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.552940    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6393d03c-e423-4037-9aef-88430e0674d1-kube-api-access-t9sbc\" (UniqueName: \"kubernetes.io/projected/6393d03c-e423-4037-9aef-88430e0674d1-kube-api-access-t9sbc\") pod \"6393d03c-e423-4037-9aef-88430e0674d1\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") "
Oct 30 21:07:50.552978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.552968    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-model-cache\") pod \"6393d03c-e423-4037-9aef-88430e0674d1\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") "
Oct 30 21:07:50.553084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.553052    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-dshm\") pod \"6393d03c-e423-4037-9aef-88430e0674d1\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") "
Oct 30 21:07:50.553147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.553105    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-home" pod "6393d03c-e423-4037-9aef-88430e0674d1" (UID: "6393d03c-e423-4037-9aef-88430e0674d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:50.553202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.553192    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-tmp-dir\") pod \"6393d03c-e423-4037-9aef-88430e0674d1\" (UID: \"6393d03c-e423-4037-9aef-88430e0674d1\") "
Oct 30 21:07:50.553281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.553247    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-model-cache" pod "6393d03c-e423-4037-9aef-88430e0674d1" (UID: "6393d03c-e423-4037-9aef-88430e0674d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:50.553454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.553427    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-tmp-dir" pod "6393d03c-e423-4037-9aef-88430e0674d1" (UID: "6393d03c-e423-4037-9aef-88430e0674d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:50.553849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.553826    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:50.553981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.553871    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:50.553981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.553887    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:50.555134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.555107    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-dshm" pod "6393d03c-e423-4037-9aef-88430e0674d1" (UID: "6393d03c-e423-4037-9aef-88430e0674d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:50.555219 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.555205    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6393d03c-e423-4037-9aef-88430e0674d1-tls-certs" pod "6393d03c-e423-4037-9aef-88430e0674d1" (UID: "6393d03c-e423-4037-9aef-88430e0674d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:50.555291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.555274    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6393d03c-e423-4037-9aef-88430e0674d1-kube-api-access-t9sbc" pod "6393d03c-e423-4037-9aef-88430e0674d1" (UID: "6393d03c-e423-4037-9aef-88430e0674d1"). InnerVolumeSpecName "kube-api-access-t9sbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:50.655320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.655292    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6393d03c-e423-4037-9aef-88430e0674d1-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:50.655320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.655318    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9sbc\" (UniqueName: \"kubernetes.io/projected/6393d03c-e423-4037-9aef-88430e0674d1-kube-api-access-t9sbc\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:50.655497 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.655328    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6393d03c-e423-4037-9aef-88430e0674d1-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:50.794123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.794039    4810 generic.go:356] "Generic (PLEG): container finished" podID="6393d03c-e423-4037-9aef-88430e0674d1" containerID="a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653" exitCode=0
Oct 30 21:07:50.794624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.794142    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"
Oct 30 21:07:50.794624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.794187    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj" event={"ID":"6393d03c-e423-4037-9aef-88430e0674d1","Type":"ContainerDied","Data":"a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653"}
Oct 30 21:07:50.794624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.794237    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj" event={"ID":"6393d03c-e423-4037-9aef-88430e0674d1","Type":"ContainerDied","Data":"712d86cd9a4f8e04aa426f6187721bdd70ab3b2a1ad486534f03a0800476eb68"}
Oct 30 21:07:50.794624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.794259    4810 scope.go:122] "RemoveContainer" containerID="a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653"
Oct 30 21:07:50.815848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.812924    4810 scope.go:122] "RemoveContainer" containerID="a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653"
Oct 30 21:07:50.815848 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:50.815256    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653\": container with ID starting with a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653 not found: ID does not exist" containerID="a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653"
Oct 30 21:07:50.815848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.815298    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653"} err="failed to get container status \"a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653\": rpc error: code = NotFound desc = could not find container \"a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653\": container with ID starting with a01be580135ccf5e994b096310eed697ad9d90748495505c437722cf7db0f653 not found: ID does not exist"
Oct 30 21:07:50.825091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.825069    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"]
Oct 30 21:07:50.828898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.828835    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-321ccd/auth-test-model-321ccd-kserve-549467c589-2nncj"]
Oct 30 21:07:50.917934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:50.917897    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"]
Oct 30 21:07:50.959195 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:50.959160    4810 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-self-signed-certs: secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:07:50.959373 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:50.959254    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs podName:0b222375-55c0-45ad-807a-4e28d4c4399a nodeName:}" failed. No retries permitted until 2026-10-30 21:07:51.459228242 +0000 UTC m=+1837.951044782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs") pod "noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a") : secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:07:51.522282 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:51.522248    4810 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-self-signed-certs: secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:07:51.522510 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:51.522327    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs podName:0b222375-55c0-45ad-807a-4e28d4c4399a nodeName:}" failed. No retries permitted until 2026-10-30 21:07:52.522309478 +0000 UTC m=+1839.014126019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs") pod "noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a") : secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:07:51.664653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:51.664617    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6393d03c-e423-4037-9aef-88430e0674d1" path="/var/lib/kubelet/pods/6393d03c-e423-4037-9aef-88430e0674d1/volumes"
Oct 30 21:07:52.539455 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:52.539415    4810 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-self-signed-certs: secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:07:52.539867 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:52.539524    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs podName:0b222375-55c0-45ad-807a-4e28d4c4399a nodeName:}" failed. No retries permitted until 2026-10-30 21:07:54.539499494 +0000 UTC m=+1841.031316061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs") pod "noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a") : secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:07:53.566881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:53.566149    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52" podUID="d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" containerName="main" containerID="cri-o://4e940f41620c92d484cebca5df5d95fd1ff1d240844cc738819b9995d06fcf21" gracePeriod=45
Oct 30 21:07:53.851928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:53.851632    4810 generic.go:356] "Generic (PLEG): container finished" podID="d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" containerID="4e940f41620c92d484cebca5df5d95fd1ff1d240844cc738819b9995d06fcf21" exitCode=0
Oct 30 21:07:53.851928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:53.851882    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52" event={"ID":"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8","Type":"ContainerDied","Data":"4e940f41620c92d484cebca5df5d95fd1ff1d240844cc738819b9995d06fcf21"}
Oct 30 21:07:53.898809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:53.898780    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:07:53.983880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:53.983006    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-dshm\") pod \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") "
Oct 30 21:07:53.983880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:53.983139    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-kube-api-access-kc7kn\" (UniqueName: \"kubernetes.io/projected/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-kube-api-access-kc7kn\") pod \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") "
Oct 30 21:07:53.983880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:53.983188    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-home\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-home\") pod \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") "
Oct 30 21:07:53.983880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:53.983249    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tmp-dir\") pod \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") "
Oct 30 21:07:53.983880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:53.983441    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tls-certs\" (UniqueName: \"kubernetes.io/secret/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tls-certs\") pod \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") "
Oct 30 21:07:53.983880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:53.983494    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-model-cache\") pod \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\" (UID: \"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8\") "
Oct 30 21:07:54.029747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.029702    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-home" pod "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" (UID: "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:54.051743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.034085    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tmp-dir" pod "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" (UID: "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:54.051743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.035055    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"]
Oct 30 21:07:54.051743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.036453    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6393d03c-e423-4037-9aef-88430e0674d1" containerName="main"
Oct 30 21:07:54.051743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.036473    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6393d03c-e423-4037-9aef-88430e0674d1" containerName="main"
Oct 30 21:07:54.051743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.036556    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" containerName="main"
Oct 30 21:07:54.051743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.036564    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" containerName="main"
Oct 30 21:07:54.051743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.036718    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6393d03c-e423-4037-9aef-88430e0674d1" containerName="main"
Oct 30 21:07:54.051743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.036741    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" containerName="main"
Oct 30 21:07:54.051743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.039003    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-model-cache" pod "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" (UID: "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:54.055001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.054049    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.056734 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.056705    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"]
Oct 30 21:07:54.059409 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.059375    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tls-certs" pod "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" (UID: "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:54.059663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.059628    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-kube-api-access-kc7kn" pod "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" (UID: "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8"). InnerVolumeSpecName "kube-api-access-kc7kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:54.061109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.061091    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-dshm" pod "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" (UID: "d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:54.064507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.064486    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a574c2\"/\"agc-cred-5b56e56cd6cc\""
Oct 30 21:07:54.064659 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.064644    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a574c2\"/\"openshift-service-ca.crt\""
Oct 30 21:07:54.065192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.064724    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a574c2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a574c2\""
Oct 30 21:07:54.065609 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.065569    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a574c2\"/\"routing-overlay\""
Oct 30 21:07:54.065692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.065658    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a574c2\"/\"payload-processing-external-model-e2e--39e93dfe-dockercfg-tmhmq\""
Oct 30 21:07:54.065829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.065813    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a574c2\"/\"kube-root-ca.crt\""
Oct 30 21:07:54.071571 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.071554    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"]
Oct 30 21:07:54.078131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.077617    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87efb90c-hsv4r"]
Oct 30 21:07:54.085622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.085312    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:54.085622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.085341    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:54.085622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.085356    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:54.085622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.085371    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc7kn\" (UniqueName: \"kubernetes.io/projected/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-kube-api-access-kc7kn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:54.085622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.085387    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:54.085622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.085406    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:54.140607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.140579    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"]
Oct 30 21:07:54.146010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.145987    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.156241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.153555    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"]
Oct 30 21:07:54.169325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.169299    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:07:54.169452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.169415    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:07:54.186497 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.186456    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b06f71-46jh2\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.186759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.186738    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.186962 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.186938    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.187165 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.187142    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjz6f\" (UniqueName: \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-kube-api-access-xjz6f\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.187362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.187339    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.188804 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.188783    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhc6\" (UniqueName: \"kubernetes.io/projected/d8a9430a-1874-4a1f-b118-f61e2d57f202-kube-api-access-ghhc6\") pod \"maas-api-revoke-sub-30b06f71-46jh2\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.189036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.188989    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b06f71-46jh2\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.291292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.291252    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.293487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.293456    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhc6\" (UniqueName: \"kubernetes.io/projected/d8a9430a-1874-4a1f-b118-f61e2d57f202-kube-api-access-ghhc6\") pod \"maas-api-revoke-sub-30b06f71-46jh2\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.293714 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.293695    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b06f71-46jh2\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.293919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.293899    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b06f71-46jh2\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.295374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.295353    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.296718 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.296692    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.297787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.297766    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjz6f\" (UniqueName: \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-kube-api-access-xjz6f\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.298457 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.297597    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.298714 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.296599    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.298823 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.295268    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b06f71-46jh2\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.335879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.327547    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b06f71-46jh2\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.335879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.330241    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhc6\" (UniqueName: \"kubernetes.io/projected/d8a9430a-1874-4a1f-b118-f61e2d57f202-kube-api-access-ghhc6\") pod \"maas-api-revoke-sub-30b06f71-46jh2\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.363879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.342797    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.364777 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.364716    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjz6f\" (UniqueName: \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-kube-api-access-xjz6f\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.406513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.406156    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.464204 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.464172    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:54.603360 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:54.603324    4810 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-self-signed-certs: secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:07:54.603779 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:54.603405    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs podName:0b222375-55c0-45ad-807a-4e28d4c4399a nodeName:}" failed. No retries permitted until 2026-10-30 21:07:58.603387184 +0000 UTC m=+1845.095203723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs") pod "noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a") : secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:07:54.705380 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.705346    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"]
Oct 30 21:07:54.707018 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:54.706995    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369f4f0c_b013_4f17_b40e_6c15e995337f.slice/crio-096c73066bb6204acd7e39d04de6ff1a5b3f5cd7888291eb7060584919502391 WatchSource:0}: Error finding container 096c73066bb6204acd7e39d04de6ff1a5b3f5cd7888291eb7060584919502391: Status 404 returned error can't find the container with id 096c73066bb6204acd7e39d04de6ff1a5b3f5cd7888291eb7060584919502391
Oct 30 21:07:54.784432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.784399    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"]
Oct 30 21:07:54.788553 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:07:54.788516    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a9430a_1874_4a1f_b118_f61e2d57f202.slice/crio-2de82841df5694880e8ce8505def743fbe7c57b8da47252523ff56031cf6c916 WatchSource:0}: Error finding container 2de82841df5694880e8ce8505def743fbe7c57b8da47252523ff56031cf6c916: Status 404 returned error can't find the container with id 2de82841df5694880e8ce8505def743fbe7c57b8da47252523ff56031cf6c916
Oct 30 21:07:54.862626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.862596    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2" event={"ID":"d8a9430a-1874-4a1f-b118-f61e2d57f202","Type":"ContainerStarted","Data":"2de82841df5694880e8ce8505def743fbe7c57b8da47252523ff56031cf6c916"}
Oct 30 21:07:54.864291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.864260    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv" event={"ID":"369f4f0c-b013-4f17-b40e-6c15e995337f","Type":"ContainerStarted","Data":"03849b9ec858ba3b61455ee0958e68138efbf66d6102850dec0cbde895a7068a"}
Oct 30 21:07:54.864476 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.864309    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv" event={"ID":"369f4f0c-b013-4f17-b40e-6c15e995337f","Type":"ContainerStarted","Data":"096c73066bb6204acd7e39d04de6ff1a5b3f5cd7888291eb7060584919502391"}
Oct 30 21:07:54.864476 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.864333    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:07:54.865584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.865556    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"
Oct 30 21:07:54.865699 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.865562    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52" event={"ID":"d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8","Type":"ContainerDied","Data":"1f807ed4d21562f6a4e6cc5648b995e6d4de86e70cf68f51d4f58dbf1fee8426"}
Oct 30 21:07:54.865699 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.865661    4810 scope.go:122] "RemoveContainer" containerID="4e940f41620c92d484cebca5df5d95fd1ff1d240844cc738819b9995d06fcf21"
Oct 30 21:07:54.888572 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.888517    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv" podStartSLOduration=1.88850158 podStartE2EDuration="1.88850158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:54.884462571 +0000 UTC m=+1841.376279118" watchObservedRunningTime="2026-10-30 21:07:54.88850158 +0000 UTC m=+1841.380318126"
Oct 30 21:07:54.904172 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.904144    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"]
Oct 30 21:07:54.909118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:54.909092    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f64640/auth-test-model-f64640-kserve-b76d57b85-z4j52"]
Oct 30 21:07:55.159397 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:55.159360    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:07:55.663259 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:55.663230    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8" path="/var/lib/kubelet/pods/d59cf7da-1bf1-4b55-9039-1ba1fcd18ee8/volumes"
Oct 30 21:07:55.663672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:55.663657    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d4776d-e845-4fd5-bfab-0fd94598c75d" path="/var/lib/kubelet/pods/e6d4776d-e845-4fd5-bfab-0fd94598c75d/volumes"
Oct 30 21:07:55.875769 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:55.875726    4810 generic.go:356] "Generic (PLEG): container finished" podID="d8a9430a-1874-4a1f-b118-f61e2d57f202" containerID="352d0289bf108964c65ecb1e924d7fa29d304da08899b8fb61f86d105b705775" exitCode=0
Oct 30 21:07:55.875967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:55.875770    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2" event={"ID":"d8a9430a-1874-4a1f-b118-f61e2d57f202","Type":"ContainerDied","Data":"352d0289bf108964c65ecb1e924d7fa29d304da08899b8fb61f86d105b705775"}
Oct 30 21:07:57.058622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.058597    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:57.141429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.141342    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-openshift-ca\") pod \"d8a9430a-1874-4a1f-b118-f61e2d57f202\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") "
Oct 30 21:07:57.141605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.141450    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a9430a-1874-4a1f-b118-f61e2d57f202-kube-api-access-ghhc6\" (UniqueName: \"kubernetes.io/projected/d8a9430a-1874-4a1f-b118-f61e2d57f202-kube-api-access-ghhc6\") pod \"d8a9430a-1874-4a1f-b118-f61e2d57f202\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") "
Oct 30 21:07:57.141710 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.141632    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-xks-ca\") pod \"d8a9430a-1874-4a1f-b118-f61e2d57f202\" (UID: \"d8a9430a-1874-4a1f-b118-f61e2d57f202\") "
Oct 30 21:07:57.142168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.142036    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-openshift-ca" pod "d8a9430a-1874-4a1f-b118-f61e2d57f202" (UID: "d8a9430a-1874-4a1f-b118-f61e2d57f202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:57.142544 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.142515    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:57.144635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.144606    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-xks-ca" pod "d8a9430a-1874-4a1f-b118-f61e2d57f202" (UID: "d8a9430a-1874-4a1f-b118-f61e2d57f202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:57.145229 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.145199    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a9430a-1874-4a1f-b118-f61e2d57f202-kube-api-access-ghhc6" pod "d8a9430a-1874-4a1f-b118-f61e2d57f202" (UID: "d8a9430a-1874-4a1f-b118-f61e2d57f202"). InnerVolumeSpecName "kube-api-access-ghhc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:57.243623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.243587    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghhc6\" (UniqueName: \"kubernetes.io/projected/d8a9430a-1874-4a1f-b118-f61e2d57f202-kube-api-access-ghhc6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:57.243623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.243624    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a9430a-1874-4a1f-b118-f61e2d57f202-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:57.757931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.757902    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:07:57.758090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.757976    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:07:57.900328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.900292    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"
Oct 30 21:07:57.900488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.900297    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2" event={"ID":"d8a9430a-1874-4a1f-b118-f61e2d57f202","Type":"ContainerDied","Data":"2de82841df5694880e8ce8505def743fbe7c57b8da47252523ff56031cf6c916"}
Oct 30 21:07:57.900488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:07:57.900422    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de82841df5694880e8ce8505def743fbe7c57b8da47252523ff56031cf6c916"
Oct 30 21:07:58.669879 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:58.669202    4810 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-self-signed-certs: secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:07:58.669879 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:07:58.669285    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs podName:0b222375-55c0-45ad-807a-4e28d4c4399a nodeName:}" failed. No retries permitted until 2026-10-30 21:08:06.669266379 +0000 UTC m=+1853.161082921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs") pod "noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a") : secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:08:01.617805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:01.617764    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl" podUID="ac0d2289-daf5-4fc4-92bc-ff73548a2d73" containerName="main" containerID="cri-o://bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc" gracePeriod=45
Oct 30 21:08:01.902019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:01.901989    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:08:01.946039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:01.945976    4810 generic.go:356] "Generic (PLEG): container finished" podID="ac0d2289-daf5-4fc4-92bc-ff73548a2d73" containerID="bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc" exitCode=0
Oct 30 21:08:01.946360 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:01.946329    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl" event={"ID":"ac0d2289-daf5-4fc4-92bc-ff73548a2d73","Type":"ContainerDied","Data":"bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc"}
Oct 30 21:08:01.946500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:01.946382    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl" event={"ID":"ac0d2289-daf5-4fc4-92bc-ff73548a2d73","Type":"ContainerDied","Data":"6904fccd609645d034aaa3308f39b15d9473918e0bbbd6d0d5fcdf4819065521"}
Oct 30 21:08:01.946500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:01.946404    4810 scope.go:122] "RemoveContainer" containerID="bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc"
Oct 30 21:08:01.946623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:01.946570    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"
Oct 30 21:08:01.968141 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:01.968111    4810 scope.go:122] "RemoveContainer" containerID="bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc"
Oct 30 21:08:01.968527 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:08:01.968504    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc\": container with ID starting with bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc not found: ID does not exist" containerID="bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc"
Oct 30 21:08:01.968614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:01.968541    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc"} err="failed to get container status \"bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc\": rpc error: code = NotFound desc = could not find container \"bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc\": container with ID starting with bec3f11305b0c248f5da20911306e85bc5fe4215fb80679f494c53d07598eccc not found: ID does not exist"
Oct 30 21:08:02.023053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.023021    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-home\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-home\") pod \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") "
Oct 30 21:08:02.023212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.023072    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-model-cache\") pod \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") "
Oct 30 21:08:02.023212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.023163    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tmp-dir\") pod \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") "
Oct 30 21:08:02.023338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.023319    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tls-certs\") pod \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") "
Oct 30 21:08:02.023423 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.023364    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-dshm\") pod \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") "
Oct 30 21:08:02.023423 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.023360    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-home" pod "ac0d2289-daf5-4fc4-92bc-ff73548a2d73" (UID: "ac0d2289-daf5-4fc4-92bc-ff73548a2d73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:02.023423 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.023372    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-model-cache" pod "ac0d2289-daf5-4fc4-92bc-ff73548a2d73" (UID: "ac0d2289-daf5-4fc4-92bc-ff73548a2d73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:02.023568 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.023455    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tmp-dir" pod "ac0d2289-daf5-4fc4-92bc-ff73548a2d73" (UID: "ac0d2289-daf5-4fc4-92bc-ff73548a2d73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:02.023568 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.023511    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-kube-api-access-mmljl\" (UniqueName: \"kubernetes.io/projected/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-kube-api-access-mmljl\") pod \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\" (UID: \"ac0d2289-daf5-4fc4-92bc-ff73548a2d73\") "
Oct 30 21:08:02.024083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.024061    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:02.024083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.024082    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:02.024282 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.024092    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:02.025695 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.025670    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-dshm" pod "ac0d2289-daf5-4fc4-92bc-ff73548a2d73" (UID: "ac0d2289-daf5-4fc4-92bc-ff73548a2d73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:02.025810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.025736    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tls-certs" pod "ac0d2289-daf5-4fc4-92bc-ff73548a2d73" (UID: "ac0d2289-daf5-4fc4-92bc-ff73548a2d73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:02.025810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.025765    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-kube-api-access-mmljl" pod "ac0d2289-daf5-4fc4-92bc-ff73548a2d73" (UID: "ac0d2289-daf5-4fc4-92bc-ff73548a2d73"). InnerVolumeSpecName "kube-api-access-mmljl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:02.125569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.125481    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:02.125569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.125521    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:02.125569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.125533    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmljl\" (UniqueName: \"kubernetes.io/projected/ac0d2289-daf5-4fc4-92bc-ff73548a2d73-kube-api-access-mmljl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:02.275937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.275902    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"]
Oct 30 21:08:02.277693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:02.277658    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-89d993/praxis-route-89d993-kserve-9b456cf57-hrrxl"]
Oct 30 21:08:03.663296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:03.663264    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0d2289-daf5-4fc4-92bc-ff73548a2d73" path="/var/lib/kubelet/pods/ac0d2289-daf5-4fc4-92bc-ff73548a2d73/volumes"
Oct 30 21:08:05.880412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:05.880382    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:08:06.412880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.411733    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"]
Oct 30 21:08:06.417876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.413393    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ac0d2289-daf5-4fc4-92bc-ff73548a2d73" containerName="main"
Oct 30 21:08:06.417876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.413420    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0d2289-daf5-4fc4-92bc-ff73548a2d73" containerName="main"
Oct 30 21:08:06.417876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.413463    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d8a9430a-1874-4a1f-b118-f61e2d57f202" containerName="revoke-keys"
Oct 30 21:08:06.417876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.413471    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a9430a-1874-4a1f-b118-f61e2d57f202" containerName="revoke-keys"
Oct 30 21:08:06.417876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.413845    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0d2289-daf5-4fc4-92bc-ff73548a2d73" containerName="main"
Oct 30 21:08:06.417876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.413904    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a9430a-1874-4a1f-b118-f61e2d57f202" containerName="revoke-keys"
Oct 30 21:08:06.421877 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.419089    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.426776 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.425583    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a50675-openshift-default-dockercfg-4pqn7\""
Oct 30 21:08:06.439876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.436460    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"]
Oct 30 21:08:06.503667 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.503626    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-credential-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.504080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.504055    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c736a64f-5235-46b3-8981-e4139e7e164c-istiod-ca-cert\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.504262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.504245    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-envoy\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.504452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.504436    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq7ph\" (UniqueName: \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-kube-api-access-lq7ph\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.504684 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.504667    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-certs\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.504835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.504820    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-data\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.505030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.505003    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.505171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.505158    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c736a64f-5235-46b3-8981-e4139e7e164c-istio-podinfo\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.505344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.505331    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-istio-token\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.606579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.606544    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-credential-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.606579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.606592    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c736a64f-5235-46b3-8981-e4139e7e164c-istiod-ca-cert\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.606844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.606619    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-envoy\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.606844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.606667    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq7ph\" (UniqueName: \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-kube-api-access-lq7ph\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.606844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.606746    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-certs\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.606844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.606773    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-data\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.606844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.606804    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.606844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.606840    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c736a64f-5235-46b3-8981-e4139e7e164c-istio-podinfo\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.607192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.606927    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-istio-token\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.607192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.607020    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-credential-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.607310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.607264    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-data\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.607486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.607454    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-certs\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.607486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.607464    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c736a64f-5235-46b3-8981-e4139e7e164c-istiod-ca-cert\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.607771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.607733    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.609237 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.609211    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-envoy\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.609777 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.609756    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c736a64f-5235-46b3-8981-e4139e7e164c-istio-podinfo\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.615578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.615552    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-istio-token\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.616044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.616027    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq7ph\" (UniqueName: \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-kube-api-access-lq7ph\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:06.708628 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:08:06.708548    4810 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-self-signed-certs: secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:08:06.708781 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:08:06.708634    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs podName:0b222375-55c0-45ad-807a-4e28d4c4399a nodeName:}" failed. No retries permitted until 2026-10-30 21:08:22.708614178 +0000 UTC m=+1869.200430718 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs") pod "noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a") : secret "noten-model-6538cc-kserve-self-signed-certs" not found
Oct 30 21:08:06.739936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:06.739899    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:07.041134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:07.041103    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"]
Oct 30 21:08:07.042125 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:08:07.042100    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc736a64f_5235_46b3_8981_e4139e7e164c.slice/crio-6e93ce2e934adce1e40b37463a84ca60f8fa2182b8afdbc045c37a0dfd05f7e2 WatchSource:0}: Error finding container 6e93ce2e934adce1e40b37463a84ca60f8fa2182b8afdbc045c37a0dfd05f7e2: Status 404 returned error can't find the container with id 6e93ce2e934adce1e40b37463a84ca60f8fa2182b8afdbc045c37a0dfd05f7e2
Oct 30 21:08:07.044995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:07.044962    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:08:07.045119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:07.045040    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:08:07.045119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:07.045099    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:08:07.845255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:07.845218    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" podUID="0b222375-55c0-45ad-807a-4e28d4c4399a" containerName="main" containerID="cri-o://f34511a029bcf87df8b58c221dbb3690aefb9d91e17b8a48248306d77cf78e8d" gracePeriod=45
Oct 30 21:08:08.016927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.016886    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" event={"ID":"c736a64f-5235-46b3-8981-e4139e7e164c","Type":"ContainerStarted","Data":"344dff5d15669b63be50c58595f1c0ef947a8330051e6b7a2c291ca735e9caf4"}
Oct 30 21:08:08.017150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.016948    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" event={"ID":"c736a64f-5235-46b3-8981-e4139e7e164c","Type":"ContainerStarted","Data":"6e93ce2e934adce1e40b37463a84ca60f8fa2182b8afdbc045c37a0dfd05f7e2"}
Oct 30 21:08:08.019336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.019300    4810 generic.go:356] "Generic (PLEG): container finished" podID="0b222375-55c0-45ad-807a-4e28d4c4399a" containerID="f34511a029bcf87df8b58c221dbb3690aefb9d91e17b8a48248306d77cf78e8d" exitCode=0
Oct 30 21:08:08.019492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.019374    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" event={"ID":"0b222375-55c0-45ad-807a-4e28d4c4399a","Type":"ContainerDied","Data":"f34511a029bcf87df8b58c221dbb3690aefb9d91e17b8a48248306d77cf78e8d"}
Oct 30 21:08:08.042331 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.042286    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" podStartSLOduration=2.042274672 podStartE2EDuration="2.042274672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:08.038116028 +0000 UTC m=+1854.529932571" watchObservedRunningTime="2026-10-30 21:08:08.042274672 +0000 UTC m=+1854.534091215"
Oct 30 21:08:08.224000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.223977    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:08:08.340135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.340092    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs\") pod \"0b222375-55c0-45ad-807a-4e28d4c4399a\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") "
Oct 30 21:08:08.340333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.340200    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-tmp-dir\") pod \"0b222375-55c0-45ad-807a-4e28d4c4399a\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") "
Oct 30 21:08:08.340333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.340226    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-dshm\") pod \"0b222375-55c0-45ad-807a-4e28d4c4399a\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") "
Oct 30 21:08:08.340428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.340339    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b222375-55c0-45ad-807a-4e28d4c4399a-kube-api-access-pdmtp\" (UniqueName: \"kubernetes.io/projected/0b222375-55c0-45ad-807a-4e28d4c4399a-kube-api-access-pdmtp\") pod \"0b222375-55c0-45ad-807a-4e28d4c4399a\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") "
Oct 30 21:08:08.340428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.340386    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-home\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-home\") pod \"0b222375-55c0-45ad-807a-4e28d4c4399a\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") "
Oct 30 21:08:08.340428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.340405    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-model-cache\") pod \"0b222375-55c0-45ad-807a-4e28d4c4399a\" (UID: \"0b222375-55c0-45ad-807a-4e28d4c4399a\") "
Oct 30 21:08:08.340580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.340429    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-tmp-dir" pod "0b222375-55c0-45ad-807a-4e28d4c4399a" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:08.340669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.340632    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-home" pod "0b222375-55c0-45ad-807a-4e28d4c4399a" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:08.340748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.340725    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-model-cache" pod "0b222375-55c0-45ad-807a-4e28d4c4399a" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:08.341075 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.341049    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:08.341197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.341080    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:08.341197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.341095    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:08.342404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.342383    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs" pod "0b222375-55c0-45ad-807a-4e28d4c4399a" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:08.342725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.342705    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-dshm" pod "0b222375-55c0-45ad-807a-4e28d4c4399a" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:08.342775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.342737    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b222375-55c0-45ad-807a-4e28d4c4399a-kube-api-access-pdmtp" pod "0b222375-55c0-45ad-807a-4e28d4c4399a" (UID: "0b222375-55c0-45ad-807a-4e28d4c4399a"). InnerVolumeSpecName "kube-api-access-pdmtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:08.442262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.442181    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b222375-55c0-45ad-807a-4e28d4c4399a-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:08.442262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.442213    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b222375-55c0-45ad-807a-4e28d4c4399a-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:08.442262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.442223    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdmtp\" (UniqueName: \"kubernetes.io/projected/0b222375-55c0-45ad-807a-4e28d4c4399a-kube-api-access-pdmtp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:08.740398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.740317    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:08.742420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.742391    4810 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:08:08.742568 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:08.742440    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:08:09.031757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:09.031677    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"
Oct 30 21:08:09.031757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:09.031687    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6" event={"ID":"0b222375-55c0-45ad-807a-4e28d4c4399a","Type":"ContainerDied","Data":"6809a908fcb3148e688dfce4df51cd5e1fc15c41ae4a260c5ee95677e005a8ee"}
Oct 30 21:08:09.031757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:09.031741    4810 scope.go:122] "RemoveContainer" containerID="f34511a029bcf87df8b58c221dbb3690aefb9d91e17b8a48248306d77cf78e8d"
Oct 30 21:08:09.063151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:09.063124    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"]
Oct 30 21:08:09.067078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:09.067055    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-6538cc/noten-model-6538cc-kserve-6c9c4c44fc-zdhh6"]
Oct 30 21:08:09.122230 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:08:09.122196    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b222375_55c0_45ad_807a_4e28d4c4399a.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:08:09.663414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:09.663386    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b222375-55c0-45ad-807a-4e28d4c4399a" path="/var/lib/kubelet/pods/0b222375-55c0-45ad-807a-4e28d4c4399a/volumes"
Oct 30 21:08:09.740372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:09.740334    4810 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:08:09.740532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:09.740400    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:08:10.740184 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:10.740148    4810 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:08:10.740602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:10.740201    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:08:11.740383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:11.740346    4810 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:08:11.740765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:11.740408    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:08:12.255169 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:12.251773    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"]
Oct 30 21:08:12.740474 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:12.740435    4810 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:08:12.741027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:12.740494    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:08:13.744901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:13.741373    4810 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:08:13.744901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:13.741427    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:08:14.745264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:14.745227    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:14.745774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:14.745752    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:14.746425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:14.746400    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:08:16.017956 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:16.017912    4810 scope.go:122] "RemoveContainer" containerID="4843d126995e2dc45db141a686f6b078ec9711befd0b6dfa67fa3d63412a83da"
Oct 30 21:08:16.029106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:16.029080    4810 scope.go:122] "RemoveContainer" containerID="d8a03a9254d846801ed105a5561f2af3d93ba6c6f82b7337234d109d74177060"
Oct 30 21:08:16.039456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:16.039429    4810 scope.go:122] "RemoveContainer" containerID="d6155ce24af3a289576e2b8a9b812f0431d49d7e0ab4876cf38704ea09282e41"
Oct 30 21:08:16.050653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:16.050631    4810 scope.go:122] "RemoveContainer" containerID="163b1137ee020ea0d299c420857e5a7e7dfd7d3d38e9a1a85e79da628fcf9619"
Oct 30 21:08:16.068547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:16.068524    4810 scope.go:122] "RemoveContainer" containerID="807e6b0dc413ee0f4b5bf46bcf10c25897faea71bd5ac6d778a291364fd15dbc"
Oct 30 21:08:16.084066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:16.084043    4810 scope.go:122] "RemoveContainer" containerID="090530abd905ef902823925b944d3a4e85aea003a0bd3821bf254ba1233ea945"
Oct 30 21:08:16.097403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:16.097381    4810 scope.go:122] "RemoveContainer" containerID="fc1314367c9ea2c0f291f7a0f1c1d2be1df3450f3cbf47dd9292679099bf027a"
Oct 30 21:08:16.110267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:16.110244    4810 scope.go:122] "RemoveContainer" containerID="6524f9a9c819ca151e7638677faab922d7a1e0fdda1e602c13e84610e518d247"
Oct 30 21:08:16.124952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:16.124692    4810 scope.go:122] "RemoveContainer" containerID="f05126a1704e9d1000259581e7e0b2a50b93f0765ea1d144b9d39b8018981c68"
Oct 30 21:08:18.600155 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.600122    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"]
Oct 30 21:08:18.600916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.600899    4810 cpu_manager.go:414] "containerMap: removing container" podUID="0b222375-55c0-45ad-807a-4e28d4c4399a" containerName="main"
Oct 30 21:08:18.600916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.600916    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b222375-55c0-45ad-807a-4e28d4c4399a" containerName="main"
Oct 30 21:08:18.601047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.601038    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b222375-55c0-45ad-807a-4e28d4c4399a" containerName="main"
Oct 30 21:08:18.608730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.608710    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:08:18.614846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.614820    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"]
Oct 30 21:08:18.648907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.638096    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-435414-dockercfg-4777h\""
Oct 30 21:08:18.648907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.638208    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-435414\""
Oct 30 21:08:18.698323 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.698286    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc395116-6963-4b0e-b42e-38336983afdf-plugins-config-volume\") pod \"payload-processing-e2e-autores-435414-6f9b5df87-kwdfk\" (UID: \"bc395116-6963-4b0e-b42e-38336983afdf\") " pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:08:18.698795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.698767    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fppft\" (UniqueName: \"kubernetes.io/projected/bc395116-6963-4b0e-b42e-38336983afdf-kube-api-access-fppft\") pod \"payload-processing-e2e-autores-435414-6f9b5df87-kwdfk\" (UID: \"bc395116-6963-4b0e-b42e-38336983afdf\") " pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:08:18.827882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.819413    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc395116-6963-4b0e-b42e-38336983afdf-plugins-config-volume\") pod \"payload-processing-e2e-autores-435414-6f9b5df87-kwdfk\" (UID: \"bc395116-6963-4b0e-b42e-38336983afdf\") " pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:08:18.827882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.819623    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fppft\" (UniqueName: \"kubernetes.io/projected/bc395116-6963-4b0e-b42e-38336983afdf-kube-api-access-fppft\") pod \"payload-processing-e2e-autores-435414-6f9b5df87-kwdfk\" (UID: \"bc395116-6963-4b0e-b42e-38336983afdf\") " pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:08:18.827882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.821538    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc395116-6963-4b0e-b42e-38336983afdf-plugins-config-volume\") pod \"payload-processing-e2e-autores-435414-6f9b5df87-kwdfk\" (UID: \"bc395116-6963-4b0e-b42e-38336983afdf\") " pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:08:18.891418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.890059    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fppft\" (UniqueName: \"kubernetes.io/projected/bc395116-6963-4b0e-b42e-38336983afdf-kube-api-access-fppft\") pod \"payload-processing-e2e-autores-435414-6f9b5df87-kwdfk\" (UID: \"bc395116-6963-4b0e-b42e-38336983afdf\") " pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:08:18.967938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:18.967902    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:08:19.276831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:19.276798    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"]
Oct 30 21:08:20.161775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:20.161737    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk" event={"ID":"bc395116-6963-4b0e-b42e-38336983afdf","Type":"ContainerStarted","Data":"f2bab10bdca213f6ab125dfb817a49c2b1a60e534ae166f0307895f549b6b03b"}
Oct 30 21:08:20.162196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:20.161800    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk" event={"ID":"bc395116-6963-4b0e-b42e-38336983afdf","Type":"ContainerStarted","Data":"ef1d7081dd805f875fe44ac9ca4be3ea1c682379fbb8bb8fe2706d8820a0ee9a"}
Oct 30 21:08:20.162196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:20.161839    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:08:20.183593 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:20.183536    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk" podStartSLOduration=2.183519959 podStartE2EDuration="2.183519959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:20.179799724 +0000 UTC m=+1866.671616270" watchObservedRunningTime="2026-10-30 21:08:20.183519959 +0000 UTC m=+1866.675336507"
Oct 30 21:08:25.158108 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:08:25.158073    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:08:26.778681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:26.778638    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"]
Oct 30 21:08:26.787109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:26.786828    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:08:26.798800 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:26.798766    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"]
Oct 30 21:08:26.852880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:26.849127    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e07442\""
Oct 30 21:08:26.852880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:26.849340    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e07442-dockercfg-xmpqq\""
Oct 30 21:08:26.912884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:26.910009    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b258a27c-5663-494e-abff-b1979edfee61-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8\" (UID: \"b258a27c-5663-494e-abff-b1979edfee61\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:08:26.929880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:26.923557    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k75wn\" (UniqueName: \"kubernetes.io/projected/b258a27c-5663-494e-abff-b1979edfee61-kube-api-access-k75wn\") pod \"payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8\" (UID: \"b258a27c-5663-494e-abff-b1979edfee61\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:08:27.047113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.047023    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b258a27c-5663-494e-abff-b1979edfee61-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8\" (UID: \"b258a27c-5663-494e-abff-b1979edfee61\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:08:27.047300 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.047147    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k75wn\" (UniqueName: \"kubernetes.io/projected/b258a27c-5663-494e-abff-b1979edfee61-kube-api-access-k75wn\") pod \"payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8\" (UID: \"b258a27c-5663-494e-abff-b1979edfee61\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:08:27.047913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.047886    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b258a27c-5663-494e-abff-b1979edfee61-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8\" (UID: \"b258a27c-5663-494e-abff-b1979edfee61\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:08:27.060911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.059824    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k75wn\" (UniqueName: \"kubernetes.io/projected/b258a27c-5663-494e-abff-b1979edfee61-kube-api-access-k75wn\") pod \"payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8\" (UID: \"b258a27c-5663-494e-abff-b1979edfee61\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:08:27.118799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.118764    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:08:27.408818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.408787    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"]
Oct 30 21:08:27.409797 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:08:27.409770    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb258a27c_5663_494e_abff_b1979edfee61.slice/crio-4460f72d098b0aac39a90794481e3f48df9ad8da6ce72f62751bea6da90ed566 WatchSource:0}: Error finding container 4460f72d098b0aac39a90794481e3f48df9ad8da6ce72f62751bea6da90ed566: Status 404 returned error can't find the container with id 4460f72d098b0aac39a90794481e3f48df9ad8da6ce72f62751bea6da90ed566
Oct 30 21:08:27.464132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.464089    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57" podUID="52e8d99c-d846-443a-a36b-98bb6e769324" containerName="main" containerID="cri-o://bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b" gracePeriod=45
Oct 30 21:08:27.783920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.783892    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:08:27.865496 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.865461    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52e8d99c-d846-443a-a36b-98bb6e769324-tls-certs\" (UniqueName: \"kubernetes.io/secret/52e8d99c-d846-443a-a36b-98bb6e769324-tls-certs\") pod \"52e8d99c-d846-443a-a36b-98bb6e769324\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") "
Oct 30 21:08:27.865656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.865626    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-dshm\") pod \"52e8d99c-d846-443a-a36b-98bb6e769324\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") "
Oct 30 21:08:27.865697 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.865684    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-tmp-dir\") pod \"52e8d99c-d846-443a-a36b-98bb6e769324\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") "
Oct 30 21:08:27.865734 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.865705    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e8d99c-d846-443a-a36b-98bb6e769324-kube-api-access-qtm76\" (UniqueName: \"kubernetes.io/projected/52e8d99c-d846-443a-a36b-98bb6e769324-kube-api-access-qtm76\") pod \"52e8d99c-d846-443a-a36b-98bb6e769324\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") "
Oct 30 21:08:27.865785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.865751    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-home\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-home\") pod \"52e8d99c-d846-443a-a36b-98bb6e769324\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") "
Oct 30 21:08:27.865845 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.865788    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-model-cache\") pod \"52e8d99c-d846-443a-a36b-98bb6e769324\" (UID: \"52e8d99c-d846-443a-a36b-98bb6e769324\") "
Oct 30 21:08:27.866018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.865984    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-tmp-dir" pod "52e8d99c-d846-443a-a36b-98bb6e769324" (UID: "52e8d99c-d846-443a-a36b-98bb6e769324"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:27.866157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.866128    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-model-cache" pod "52e8d99c-d846-443a-a36b-98bb6e769324" (UID: "52e8d99c-d846-443a-a36b-98bb6e769324"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:27.866224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.866136    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-home" pod "52e8d99c-d846-443a-a36b-98bb6e769324" (UID: "52e8d99c-d846-443a-a36b-98bb6e769324"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:27.866485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.866461    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:27.866574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.866488    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:27.866574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.866503    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:27.867897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.867867    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-dshm" pod "52e8d99c-d846-443a-a36b-98bb6e769324" (UID: "52e8d99c-d846-443a-a36b-98bb6e769324"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:27.867991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.867915    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e8d99c-d846-443a-a36b-98bb6e769324-tls-certs" pod "52e8d99c-d846-443a-a36b-98bb6e769324" (UID: "52e8d99c-d846-443a-a36b-98bb6e769324"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:27.868041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.868002    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e8d99c-d846-443a-a36b-98bb6e769324-kube-api-access-qtm76" pod "52e8d99c-d846-443a-a36b-98bb6e769324" (UID: "52e8d99c-d846-443a-a36b-98bb6e769324"). InnerVolumeSpecName "kube-api-access-qtm76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:27.968303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.968223    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtm76\" (UniqueName: \"kubernetes.io/projected/52e8d99c-d846-443a-a36b-98bb6e769324-kube-api-access-qtm76\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:27.968303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.968254    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e8d99c-d846-443a-a36b-98bb6e769324-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:27.968303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:27.968263    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e8d99c-d846-443a-a36b-98bb6e769324-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:28.243890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.243795    4810 generic.go:356] "Generic (PLEG): container finished" podID="52e8d99c-d846-443a-a36b-98bb6e769324" containerID="bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b" exitCode=0
Oct 30 21:08:28.243890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.243841    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57" event={"ID":"52e8d99c-d846-443a-a36b-98bb6e769324","Type":"ContainerDied","Data":"bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b"}
Oct 30 21:08:28.243890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.243848    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"
Oct 30 21:08:28.244164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.243900    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57" event={"ID":"52e8d99c-d846-443a-a36b-98bb6e769324","Type":"ContainerDied","Data":"cd30bcc68dbe878050d297fe07024c3b30c11c8172b7862291cc3f714af4f1c7"}
Oct 30 21:08:28.244164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.243923    4810 scope.go:122] "RemoveContainer" containerID="bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b"
Oct 30 21:08:28.246073 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.246050    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8" event={"ID":"b258a27c-5663-494e-abff-b1979edfee61","Type":"ContainerStarted","Data":"bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796"}
Oct 30 21:08:28.246221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.246093    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8" event={"ID":"b258a27c-5663-494e-abff-b1979edfee61","Type":"ContainerStarted","Data":"4460f72d098b0aac39a90794481e3f48df9ad8da6ce72f62751bea6da90ed566"}
Oct 30 21:08:28.246221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.246144    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:08:28.255392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.255370    4810 scope.go:122] "RemoveContainer" containerID="bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b"
Oct 30 21:08:28.255698 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:08:28.255676    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b\": container with ID starting with bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b not found: ID does not exist" containerID="bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b"
Oct 30 21:08:28.255786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.255706    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b"} err="failed to get container status \"bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b\": rpc error: code = NotFound desc = could not find container \"bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b\": container with ID starting with bd2c7911879ddf0906cbed0983b225c856c62164103f6d1b0f0e6fdac369a02b not found: ID does not exist"
Oct 30 21:08:28.268138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.268094    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8" podStartSLOduration=2.268081649 podStartE2EDuration="2.268081649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:28.265670521 +0000 UTC m=+1874.757487067" watchObservedRunningTime="2026-10-30 21:08:28.268081649 +0000 UTC m=+1874.759898194"
Oct 30 21:08:28.283710 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.283681    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"]
Oct 30 21:08:28.288127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.288095    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57"]
Oct 30 21:08:28.401555 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.401520    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:08:28.758626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:28.758587    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-a50675/auth-test-model-a50675-kserve-78ffc6568f-rvt57" podUID="52e8d99c-d846-443a-a36b-98bb6e769324" containerName="main" probeResult="failure" output="Get \"https://10.132.0.202:8000/ready\": context deadline exceeded"
Oct 30 21:08:29.673254 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:29.673213    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e8d99c-d846-443a-a36b-98bb6e769324" path="/var/lib/kubelet/pods/52e8d99c-d846-443a-a36b-98bb6e769324/volumes"
Oct 30 21:08:31.172438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:31.172403    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:08:34.802109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.802071    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"]
Oct 30 21:08:34.802822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.802805    4810 cpu_manager.go:414] "containerMap: removing container" podUID="52e8d99c-d846-443a-a36b-98bb6e769324" containerName="main"
Oct 30 21:08:34.802912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.802823    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e8d99c-d846-443a-a36b-98bb6e769324" containerName="main"
Oct 30 21:08:34.802982 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.802947    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e8d99c-d846-443a-a36b-98bb6e769324" containerName="main"
Oct 30 21:08:34.806327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.806306    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:34.812171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.812144    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:34.812307 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.812183    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7tvb\" (UniqueName: \"kubernetes.io/projected/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-kube-api-access-p7tvb\") pod \"maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:34.812479 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.812459    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:34.817633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.817606    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"]
Oct 30 21:08:34.883646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.883615    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:08:34.883896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.883674    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:08:34.913645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.913609    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:34.913645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.913642    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7tvb\" (UniqueName: \"kubernetes.io/projected/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-kube-api-access-p7tvb\") pod \"maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:34.913931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.913758    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:34.914339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.914319    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:34.916345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.916316    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:34.922384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:34.922351    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7tvb\" (UniqueName: \"kubernetes.io/projected/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-kube-api-access-p7tvb\") pod \"maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:35.131703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:35.131669    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:35.424901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:35.424849    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"]
Oct 30 21:08:35.425186 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:08:35.425166    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f0f303_e4a6_48a4_ab7f_ece3ae7cbe25.slice/crio-80ed390abc9f5009c776b5920b9046f7ba8b4eddc3dcc7e205b85053cbd93f7c WatchSource:0}: Error finding container 80ed390abc9f5009c776b5920b9046f7ba8b4eddc3dcc7e205b85053cbd93f7c: Status 404 returned error can't find the container with id 80ed390abc9f5009c776b5920b9046f7ba8b4eddc3dcc7e205b85053cbd93f7c
Oct 30 21:08:36.330143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:36.330099    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn" event={"ID":"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25","Type":"ContainerStarted","Data":"2cc3646a85962af5aafd9601d1c8ca10cff1b0d9e155bee4aed6c5a36f6d7fce"}
Oct 30 21:08:36.330653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:36.330168    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn" event={"ID":"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25","Type":"ContainerStarted","Data":"80ed390abc9f5009c776b5920b9046f7ba8b4eddc3dcc7e205b85053cbd93f7c"}
Oct 30 21:08:36.348431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:36.348369    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn" podStartSLOduration=2.348355041 podStartE2EDuration="2.348355041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:36.345715979 +0000 UTC m=+1882.837532528" watchObservedRunningTime="2026-10-30 21:08:36.348355041 +0000 UTC m=+1882.840171576"
Oct 30 21:08:37.358339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:37.358297    4810 generic.go:356] "Generic (PLEG): container finished" podID="a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25" containerID="2cc3646a85962af5aafd9601d1c8ca10cff1b0d9e155bee4aed6c5a36f6d7fce" exitCode=0
Oct 30 21:08:37.359112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:37.359081    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn" event={"ID":"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25","Type":"ContainerDied","Data":"2cc3646a85962af5aafd9601d1c8ca10cff1b0d9e155bee4aed6c5a36f6d7fce"}
Oct 30 21:08:38.617392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:38.617315    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:38.691954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:38.691910    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-xks-ca\") pod \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") "
Oct 30 21:08:38.692575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:38.692549    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-openshift-ca\") pod \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") "
Oct 30 21:08:38.693295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:38.693131    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-openshift-ca" pod "a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25" (UID: "a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:38.693844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:38.693537    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-kube-api-access-p7tvb\" (UniqueName: \"kubernetes.io/projected/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-kube-api-access-p7tvb\") pod \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\" (UID: \"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25\") "
Oct 30 21:08:38.695062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:38.694638    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-xks-ca" pod "a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25" (UID: "a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:38.695062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:38.694743    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:38.695062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:38.694768    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:38.695998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:38.695977    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-kube-api-access-p7tvb" pod "a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25" (UID: "a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25"). InnerVolumeSpecName "kube-api-access-p7tvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:38.795690 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:38.795648    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7tvb\" (UniqueName: \"kubernetes.io/projected/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25-kube-api-access-p7tvb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:39.259150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:39.259120    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:08:39.375448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:39.375417    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"
Oct 30 21:08:39.375631 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:39.375438    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn" event={"ID":"a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25","Type":"ContainerDied","Data":"80ed390abc9f5009c776b5920b9046f7ba8b4eddc3dcc7e205b85053cbd93f7c"}
Oct 30 21:08:39.375631 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:39.375500    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ed390abc9f5009c776b5920b9046f7ba8b4eddc3dcc7e205b85053cbd93f7c"
Oct 30 21:08:42.781874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:42.781814    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"]
Oct 30 21:08:42.782365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:42.782095    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy" containerID="cri-o://0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9" gracePeriod=30
Oct 30 21:08:48.077325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.077294    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 21:08:48.167610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.167582    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-podinfo\") pod \"a8ff0f99-80ba-4272-8685-24b8c172e815\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") "
Oct 30 21:08:48.167795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.167626    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-envoy\") pod \"a8ff0f99-80ba-4272-8685-24b8c172e815\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") "
Oct 30 21:08:48.167795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.167702    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-socket\") pod \"a8ff0f99-80ba-4272-8685-24b8c172e815\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") "
Oct 30 21:08:48.167795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.167753    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-credential-socket\") pod \"a8ff0f99-80ba-4272-8685-24b8c172e815\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") "
Oct 30 21:08:48.168003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.167791    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-data\") pod \"a8ff0f99-80ba-4272-8685-24b8c172e815\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") "
Oct 30 21:08:48.168003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.167901    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-kube-api-access-vgdwj\" (UniqueName: \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-kube-api-access-vgdwj\") pod \"a8ff0f99-80ba-4272-8685-24b8c172e815\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") "
Oct 30 21:08:48.168003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.167946    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-certs\") pod \"a8ff0f99-80ba-4272-8685-24b8c172e815\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") "
Oct 30 21:08:48.168003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.167978    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-token\" (UniqueName: \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-token\") pod \"a8ff0f99-80ba-4272-8685-24b8c172e815\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") "
Oct 30 21:08:48.168231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168083    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-socket" pod "a8ff0f99-80ba-4272-8685-24b8c172e815" (UID: "a8ff0f99-80ba-4272-8685-24b8c172e815"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:48.168231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168105    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-data" pod "a8ff0f99-80ba-4272-8685-24b8c172e815" (UID: "a8ff0f99-80ba-4272-8685-24b8c172e815"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:48.168231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168121    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-credential-socket" pod "a8ff0f99-80ba-4272-8685-24b8c172e815" (UID: "a8ff0f99-80ba-4272-8685-24b8c172e815"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:48.168231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168143    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ff0f99-80ba-4272-8685-24b8c172e815-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ff0f99-80ba-4272-8685-24b8c172e815-istiod-ca-cert\") pod \"a8ff0f99-80ba-4272-8685-24b8c172e815\" (UID: \"a8ff0f99-80ba-4272-8685-24b8c172e815\") "
Oct 30 21:08:48.168452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168352    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-certs" pod "a8ff0f99-80ba-4272-8685-24b8c172e815" (UID: "a8ff0f99-80ba-4272-8685-24b8c172e815"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:48.168723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168696    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ff0f99-80ba-4272-8685-24b8c172e815-istiod-ca-cert" pod "a8ff0f99-80ba-4272-8685-24b8c172e815" (UID: "a8ff0f99-80ba-4272-8685-24b8c172e815"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:48.168918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168897    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ff0f99-80ba-4272-8685-24b8c172e815-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:48.169013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168922    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:48.169013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168938    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:48.169013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168954    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:48.169013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.168968    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:48.170162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.170072    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-envoy" pod "a8ff0f99-80ba-4272-8685-24b8c172e815" (UID: "a8ff0f99-80ba-4272-8685-24b8c172e815"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:08:48.170162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.170141    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-podinfo" pod "a8ff0f99-80ba-4272-8685-24b8c172e815" (UID: "a8ff0f99-80ba-4272-8685-24b8c172e815"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:08:48.170315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.170284    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-kube-api-access-vgdwj" pod "a8ff0f99-80ba-4272-8685-24b8c172e815" (UID: "a8ff0f99-80ba-4272-8685-24b8c172e815"). InnerVolumeSpecName "kube-api-access-vgdwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:48.170565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.170549    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-token" pod "a8ff0f99-80ba-4272-8685-24b8c172e815" (UID: "a8ff0f99-80ba-4272-8685-24b8c172e815"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:48.270183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.270146    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:48.270183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.270179    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:48.270183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.270192    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgdwj\" (UniqueName: \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-kube-api-access-vgdwj\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:48.270446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.270205    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ff0f99-80ba-4272-8685-24b8c172e815-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:48.451258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.451224    4810 generic.go:356] "Generic (PLEG): container finished" podID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerID="0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9" exitCode=0
Oct 30 21:08:48.451415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.451263    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" event={"ID":"a8ff0f99-80ba-4272-8685-24b8c172e815","Type":"ContainerDied","Data":"0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9"}
Oct 30 21:08:48.451415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.451278    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"
Oct 30 21:08:48.451415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.451322    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84" event={"ID":"a8ff0f99-80ba-4272-8685-24b8c172e815","Type":"ContainerDied","Data":"e349bedef9f81acd7e3351513c35be7345531fbeaad5c19ffb42d0cc7d937847"}
Oct 30 21:08:48.451415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.451345    4810 scope.go:122] "RemoveContainer" containerID="0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9"
Oct 30 21:08:48.463397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.463379    4810 scope.go:122] "RemoveContainer" containerID="0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9"
Oct 30 21:08:48.463757 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:08:48.463737    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9\": container with ID starting with 0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9 not found: ID does not exist" containerID="0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9"
Oct 30 21:08:48.463838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.463770    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9"} err="failed to get container status \"0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9\": rpc error: code = NotFound desc = could not find container \"0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9\": container with ID starting with 0e50e9e228530dd614e83f9775e662ba457fc681e71c00a3fba458bcbdb1c9a9 not found: ID does not exist"
Oct 30 21:08:48.477907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.477840    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"]
Oct 30 21:08:48.492222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:48.492185    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-69c84"]
Oct 30 21:08:49.663535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:49.663493    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" path="/var/lib/kubelet/pods/a8ff0f99-80ba-4272-8685-24b8c172e815/volumes"
Oct 30 21:08:55.158526 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:08:55.158486    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:08:55.617604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:55.617517    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"]
Oct 30 21:08:55.713900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:55.713853    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-49c7c2-xswxn"]
Oct 30 21:08:57.673537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:57.673504    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25" path="/var/lib/kubelet/pods/a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25/volumes"
Oct 30 21:08:59.557593 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.557539    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"]
Oct 30 21:08:59.560449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.560407    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25" containerName="revoke-keys"
Oct 30 21:08:59.560449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.560440    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25" containerName="revoke-keys"
Oct 30 21:08:59.560671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.560474    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy"
Oct 30 21:08:59.560671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.560482    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy"
Oct 30 21:08:59.560671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.560660    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f0f303-e4a6-48a4-ab7f-ece3ae7cbe25" containerName="revoke-keys"
Oct 30 21:08:59.560835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.560680    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ff0f99-80ba-4272-8685-24b8c172e815" containerName="istio-proxy"
Oct 30 21:08:59.604107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.604057    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"]
Oct 30 21:08:59.604312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.604190    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.647165 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.647132    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-435414\"/\"openshift-service-ca.crt\""
Oct 30 21:08:59.650928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.650899    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-435414\"/\"kube-root-ca.crt\""
Oct 30 21:08:59.653691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.652938    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-435414\"/\"autores-model-435414-kserve-self-signed-certs\""
Oct 30 21:08:59.653691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.653052    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-435414\"/\"default-dockercfg-fh9bg\""
Oct 30 21:08:59.675819 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.675786    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-tmp-dir\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.675819 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.675825    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15a08552-c32c-4113-ad76-d9bfb70d721e-tls-certs\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.676066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.675875    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-home\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.676115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.676086    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mzg\" (UniqueName: \"kubernetes.io/projected/15a08552-c32c-4113-ad76-d9bfb70d721e-kube-api-access-47mzg\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.676206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.676184    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-model-cache\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.676297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.676276    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-dshm\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.809319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.778415    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-tmp-dir\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.809319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.778481    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15a08552-c32c-4113-ad76-d9bfb70d721e-tls-certs\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.809319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.778657    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-home\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.809319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.778765    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mzg\" (UniqueName: \"kubernetes.io/projected/15a08552-c32c-4113-ad76-d9bfb70d721e-kube-api-access-47mzg\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.809319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.778830    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-model-cache\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.809319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.778898    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-dshm\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.809319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.780772    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-home\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.809319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.781080    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-tmp-dir\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.813981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.813946    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-model-cache\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.816929 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.816900    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-dshm\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.833233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.833194    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15a08552-c32c-4113-ad76-d9bfb70d721e-tls-certs\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.869878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.842674    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mzg\" (UniqueName: \"kubernetes.io/projected/15a08552-c32c-4113-ad76-d9bfb70d721e-kube-api-access-47mzg\") pod \"autores-model-435414-kserve-78cdfcd9f7-64tpk\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") " pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:08:59.963762 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:08:59.963716    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:09:00.274622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:00.274591    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"]
Oct 30 21:09:00.275166 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:09:00.275137    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15a08552_c32c_4113_ad76_d9bfb70d721e.slice/crio-014abd974848dd8ae90bf649a48e8b94fdecc771a98e5ce1e76a449bf4d26b2f WatchSource:0}: Error finding container 014abd974848dd8ae90bf649a48e8b94fdecc771a98e5ce1e76a449bf4d26b2f: Status 404 returned error can't find the container with id 014abd974848dd8ae90bf649a48e8b94fdecc771a98e5ce1e76a449bf4d26b2f
Oct 30 21:09:00.648677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:00.648634    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk" event={"ID":"15a08552-c32c-4113-ad76-d9bfb70d721e","Type":"ContainerStarted","Data":"eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e"}
Oct 30 21:09:00.649098 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:00.648689    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk" event={"ID":"15a08552-c32c-4113-ad76-d9bfb70d721e","Type":"ContainerStarted","Data":"014abd974848dd8ae90bf649a48e8b94fdecc771a98e5ce1e76a449bf4d26b2f"}
Oct 30 21:09:00.666368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:00.666307    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk" podStartSLOduration=1.666290821 podStartE2EDuration="1.666290821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:00.66588581 +0000 UTC m=+1907.157702357" watchObservedRunningTime="2026-10-30 21:09:00.666290821 +0000 UTC m=+1907.158107368"
Oct 30 21:09:00.964128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:00.964035    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:09:06.840258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.840219    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"]
Oct 30 21:09:06.845746 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.845717    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:06.856918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.856890    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"]
Oct 30 21:09:06.909982 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.909934    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c2eacd-60e4-4cf2-a16d-734cbf585086-istiod-ca-cert\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:06.910164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.910058    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-token\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:06.910164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.910098    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-envoy\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:06.910164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.910127    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-credential-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:06.910287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.910232    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-data\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:06.910287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.910266    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-podinfo\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:06.910460 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.910431    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6flq2\" (UniqueName: \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-kube-api-access-6flq2\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:06.910502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.910478    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-certs\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:06.910580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.910567    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:06.941552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:06.941516    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e07442-openshift-default-dockercfg-zxmtx\""
Oct 30 21:09:07.011758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.011709    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6flq2\" (UniqueName: \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-kube-api-access-6flq2\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.011970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.011775    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-certs\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.011970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.011905    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012018    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c2eacd-60e4-4cf2-a16d-734cbf585086-istiod-ca-cert\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012118    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-token\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012164    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-envoy\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012204    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-credential-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012267    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-data\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012285    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-certs\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012321    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-podinfo\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012619    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-credential-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012728    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-data\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012770    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.012814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.012779    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c2eacd-60e4-4cf2-a16d-734cbf585086-istiod-ca-cert\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.014617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.014583    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-envoy\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.015014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.014992    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-podinfo\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.021585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.021564    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-token\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.021793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.021772    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6flq2\" (UniqueName: \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-kube-api-access-6flq2\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.167401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.167368    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:07.505661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.505613    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:09:07.505800 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.505714    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:09:07.505800 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.505757    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:09:07.519954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.519918    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"]
Oct 30 21:09:07.749916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.749762    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" event={"ID":"07c2eacd-60e4-4cf2-a16d-734cbf585086","Type":"ContainerStarted","Data":"b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d"}
Oct 30 21:09:07.749916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.749847    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" event={"ID":"07c2eacd-60e4-4cf2-a16d-734cbf585086","Type":"ContainerStarted","Data":"a2fdb1e42387e818e570d18f1d055832834949507e302dc7d90a27625e9dd257"}
Oct 30 21:09:07.782193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:07.782125    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" podStartSLOduration=1.782111715 podStartE2EDuration="1.782111715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:07.78102756 +0000 UTC m=+1914.272844116" watchObservedRunningTime="2026-10-30 21:09:07.782111715 +0000 UTC m=+1914.273928259"
Oct 30 21:09:08.167909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:08.167844    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:08.169742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:08.169717    4810 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:09:08.169905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:08.169783    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 21:09:09.168396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:09.168356    4810 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:09:09.168809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:09.168441    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 21:09:09.969256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:09.964087    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:09:10.015635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:10.015594    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:09:10.167920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:10.167886    4810 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:09:10.168092 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:10.167940    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 21:09:11.043404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:11.043375    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:09:11.168826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:11.168791    4810 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:09:11.169134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:11.168883    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 21:09:12.168233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:12.168189    4810 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:09:12.168654 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:12.168257    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 21:09:13.168226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:13.168188    4810 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 21:09:13.168441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:13.168246    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 21:09:14.373780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:14.373748    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:14.374694 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:14.374668    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:14.375780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:14.375760    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:09:15.302649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.302612    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"]
Oct 30 21:09:15.785894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.785836    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"]
Oct 30 21:09:15.792665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.792626    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:15.796841 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.796816    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:09:15.797144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.797127    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:09:15.799260 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.799233    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"]
Oct 30 21:09:15.879136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.879101    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-435414-j86bx\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:15.879383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.879356    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjpwm\" (UniqueName: \"kubernetes.io/projected/191256f6-75a7-4f7a-8900-69cb82192599-kube-api-access-mjpwm\") pod \"maas-api-revoke-keys-e2e-autores-435414-j86bx\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:15.879553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.879535    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-435414-j86bx\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:15.980690 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.980653    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjpwm\" (UniqueName: \"kubernetes.io/projected/191256f6-75a7-4f7a-8900-69cb82192599-kube-api-access-mjpwm\") pod \"maas-api-revoke-keys-e2e-autores-435414-j86bx\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:15.980924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.980815    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-435414-j86bx\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:15.980924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.980914    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-435414-j86bx\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:15.981542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.981516    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-435414-j86bx\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:15.983844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.983819    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-435414-j86bx\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:15.989842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:15.989812    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjpwm\" (UniqueName: \"kubernetes.io/projected/191256f6-75a7-4f7a-8900-69cb82192599-kube-api-access-mjpwm\") pod \"maas-api-revoke-keys-e2e-autores-435414-j86bx\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:16.133212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:16.133174    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:16.271709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:16.271274    4810 scope.go:122] "RemoveContainer" containerID="760c2e0de17fdb1503f62a5a1f70d20fc514b3e5d60b0a690f563416fcbac35b"
Oct 30 21:09:16.285474 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:16.285449    4810 scope.go:122] "RemoveContainer" containerID="0137e7384c8720feb37f87a9eb03921df3d0bbdc081886a04cc3516ab749d296"
Oct 30 21:09:16.305662 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:16.305621    4810 scope.go:122] "RemoveContainer" containerID="601b88ad1b30e885f2c5fc686f6ecdb6590553a4635695a68378ce4e3280aa63"
Oct 30 21:09:16.318577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:16.318546    4810 scope.go:122] "RemoveContainer" containerID="d0bd8e5e14063d5b952ebfddf9f64a5c30b35999e3156d0d0bc6b0bdf843bfc8"
Oct 30 21:09:16.438185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:16.438146    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"]
Oct 30 21:09:16.439246 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:09:16.439221    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191256f6_75a7_4f7a_8900_69cb82192599.slice/crio-f47ac805c4b6d5144fe0371dc64d96e9c2b49bab781db46e3b7dd43c3e519f77 WatchSource:0}: Error finding container f47ac805c4b6d5144fe0371dc64d96e9c2b49bab781db46e3b7dd43c3e519f77: Status 404 returned error can't find the container with id f47ac805c4b6d5144fe0371dc64d96e9c2b49bab781db46e3b7dd43c3e519f77
Oct 30 21:09:17.004382 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:17.004341    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx" event={"ID":"191256f6-75a7-4f7a-8900-69cb82192599","Type":"ContainerStarted","Data":"56e58a906c821c0c62c435f37675eab6a654d5aa92c2d5891f3e90776fb194df"}
Oct 30 21:09:17.004803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:17.004404    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx" event={"ID":"191256f6-75a7-4f7a-8900-69cb82192599","Type":"ContainerStarted","Data":"f47ac805c4b6d5144fe0371dc64d96e9c2b49bab781db46e3b7dd43c3e519f77"}
Oct 30 21:09:17.027442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:17.027381    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx" podStartSLOduration=2.027366664 podStartE2EDuration="2.027366664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:17.024298675 +0000 UTC m=+1923.516115225" watchObservedRunningTime="2026-10-30 21:09:17.027366664 +0000 UTC m=+1923.519183208"
Oct 30 21:09:18.014126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:18.014086    4810 generic.go:356] "Generic (PLEG): container finished" podID="191256f6-75a7-4f7a-8900-69cb82192599" containerID="56e58a906c821c0c62c435f37675eab6a654d5aa92c2d5891f3e90776fb194df" exitCode=0
Oct 30 21:09:18.014547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:18.014173    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx" event={"ID":"191256f6-75a7-4f7a-8900-69cb82192599","Type":"ContainerDied","Data":"56e58a906c821c0c62c435f37675eab6a654d5aa92c2d5891f3e90776fb194df"}
Oct 30 21:09:19.267319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:19.267284    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:19.332839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:19.332748    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191256f6-75a7-4f7a-8900-69cb82192599-kube-api-access-mjpwm\" (UniqueName: \"kubernetes.io/projected/191256f6-75a7-4f7a-8900-69cb82192599-kube-api-access-mjpwm\") pod \"191256f6-75a7-4f7a-8900-69cb82192599\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") "
Oct 30 21:09:19.332839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:19.332810    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-xks-ca\") pod \"191256f6-75a7-4f7a-8900-69cb82192599\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") "
Oct 30 21:09:19.333104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:19.332974    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-openshift-ca\") pod \"191256f6-75a7-4f7a-8900-69cb82192599\" (UID: \"191256f6-75a7-4f7a-8900-69cb82192599\") "
Oct 30 21:09:19.333587 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:19.333558    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-openshift-ca" pod "191256f6-75a7-4f7a-8900-69cb82192599" (UID: "191256f6-75a7-4f7a-8900-69cb82192599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:19.335326 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:19.335286    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191256f6-75a7-4f7a-8900-69cb82192599-kube-api-access-mjpwm" pod "191256f6-75a7-4f7a-8900-69cb82192599" (UID: "191256f6-75a7-4f7a-8900-69cb82192599"). InnerVolumeSpecName "kube-api-access-mjpwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:19.335567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:19.335538    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-xks-ca" pod "191256f6-75a7-4f7a-8900-69cb82192599" (UID: "191256f6-75a7-4f7a-8900-69cb82192599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:19.435314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:19.435281    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjpwm\" (UniqueName: \"kubernetes.io/projected/191256f6-75a7-4f7a-8900-69cb82192599-kube-api-access-mjpwm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:19.435314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:19.435311    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:19.435314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:19.435321    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191256f6-75a7-4f7a-8900-69cb82192599-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:20.033422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:20.033336    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"
Oct 30 21:09:20.033422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:20.033369    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx" event={"ID":"191256f6-75a7-4f7a-8900-69cb82192599","Type":"ContainerDied","Data":"f47ac805c4b6d5144fe0371dc64d96e9c2b49bab781db46e3b7dd43c3e519f77"}
Oct 30 21:09:20.033605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:20.033454    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f47ac805c4b6d5144fe0371dc64d96e9c2b49bab781db46e3b7dd43c3e519f77"
Oct 30 21:09:25.180040 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:09:25.179959    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:09:25.991197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:25.991148    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"]
Oct 30 21:09:27.947692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.947651    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"]
Oct 30 21:09:27.948788 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.948763    4810 cpu_manager.go:414] "containerMap: removing container" podUID="191256f6-75a7-4f7a-8900-69cb82192599" containerName="revoke-keys"
Oct 30 21:09:27.948788 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.948788    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191256f6-75a7-4f7a-8900-69cb82192599" containerName="revoke-keys"
Oct 30 21:09:27.949066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.949052    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="191256f6-75a7-4f7a-8900-69cb82192599" containerName="revoke-keys"
Oct 30 21:09:27.954427 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.954408    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"]
Oct 30 21:09:27.954544 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.954526    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:09:27.959384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.959360    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:09:27.963413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.963391    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d95261-dockercfg-9s9qh\""
Oct 30 21:09:27.963413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.963396    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d95261\""
Oct 30 21:09:27.963898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.963879    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"]
Oct 30 21:09:27.968516 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:27.968488    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"]
Oct 30 21:09:28.175054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.175015    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/297f5eb4-df4e-47dc-bb9b-41505d5da2be-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z\" (UID: \"297f5eb4-df4e-47dc-bb9b-41505d5da2be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:09:28.175280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.175076    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7x5\" (UniqueName: \"kubernetes.io/projected/1c20507b-2dd0-4490-b5eb-dc475f769a04-kube-api-access-vw7x5\") pod \"payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk\" (UID: \"1c20507b-2dd0-4490-b5eb-dc475f769a04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:09:28.175354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.175310    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5bwh\" (UniqueName: \"kubernetes.io/projected/297f5eb4-df4e-47dc-bb9b-41505d5da2be-kube-api-access-z5bwh\") pod \"payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z\" (UID: \"297f5eb4-df4e-47dc-bb9b-41505d5da2be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:09:28.175457 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.175435    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c20507b-2dd0-4490-b5eb-dc475f769a04-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk\" (UID: \"1c20507b-2dd0-4490-b5eb-dc475f769a04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:09:28.235851 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.235762    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:09:28.295127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.295084    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5bwh\" (UniqueName: \"kubernetes.io/projected/297f5eb4-df4e-47dc-bb9b-41505d5da2be-kube-api-access-z5bwh\") pod \"payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z\" (UID: \"297f5eb4-df4e-47dc-bb9b-41505d5da2be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:09:28.295421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.295396    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c20507b-2dd0-4490-b5eb-dc475f769a04-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk\" (UID: \"1c20507b-2dd0-4490-b5eb-dc475f769a04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:09:28.295668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.295646    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/297f5eb4-df4e-47dc-bb9b-41505d5da2be-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z\" (UID: \"297f5eb4-df4e-47dc-bb9b-41505d5da2be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:09:28.295836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.295818    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7x5\" (UniqueName: \"kubernetes.io/projected/1c20507b-2dd0-4490-b5eb-dc475f769a04-kube-api-access-vw7x5\") pod \"payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk\" (UID: \"1c20507b-2dd0-4490-b5eb-dc475f769a04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:09:28.297454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.297430    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c20507b-2dd0-4490-b5eb-dc475f769a04-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk\" (UID: \"1c20507b-2dd0-4490-b5eb-dc475f769a04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:09:28.308523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.298322    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/297f5eb4-df4e-47dc-bb9b-41505d5da2be-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z\" (UID: \"297f5eb4-df4e-47dc-bb9b-41505d5da2be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:09:28.338899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.338524    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7x5\" (UniqueName: \"kubernetes.io/projected/1c20507b-2dd0-4490-b5eb-dc475f769a04-kube-api-access-vw7x5\") pod \"payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk\" (UID: \"1c20507b-2dd0-4490-b5eb-dc475f769a04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:09:28.340205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.340177    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5bwh\" (UniqueName: \"kubernetes.io/projected/297f5eb4-df4e-47dc-bb9b-41505d5da2be-kube-api-access-z5bwh\") pod \"payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z\" (UID: \"297f5eb4-df4e-47dc-bb9b-41505d5da2be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:09:28.489562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.489477    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:09:28.585067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.580150    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:09:28.795622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.795170    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"]
Oct 30 21:09:28.796608 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:09:28.796529    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297f5eb4_df4e_47dc_bb9b_41505d5da2be.slice/crio-09356e5fd150c45113e9c7ef3515c9856dcc8858f356294479a19dff1ade4189 WatchSource:0}: Error finding container 09356e5fd150c45113e9c7ef3515c9856dcc8858f356294479a19dff1ade4189: Status 404 returned error can't find the container with id 09356e5fd150c45113e9c7ef3515c9856dcc8858f356294479a19dff1ade4189
Oct 30 21:09:28.932901 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:09:28.932833    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c20507b_2dd0_4490_b5eb_dc475f769a04.slice/crio-3e03d5f6ef947cacb64e0f296786bdfb042d1e38113f37ca4edb1ef61cecbde3 WatchSource:0}: Error finding container 3e03d5f6ef947cacb64e0f296786bdfb042d1e38113f37ca4edb1ef61cecbde3: Status 404 returned error can't find the container with id 3e03d5f6ef947cacb64e0f296786bdfb042d1e38113f37ca4edb1ef61cecbde3
Oct 30 21:09:28.933905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:28.933880    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"]
Oct 30 21:09:29.242133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:29.242085    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk" event={"ID":"1c20507b-2dd0-4490-b5eb-dc475f769a04","Type":"ContainerStarted","Data":"db080dffaadef9ac3cb966393d7ea15490a4a43a8f039f3fbc1e22bc952f3bbf"}
Oct 30 21:09:29.242667 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:29.242161    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk" event={"ID":"1c20507b-2dd0-4490-b5eb-dc475f769a04","Type":"ContainerStarted","Data":"3e03d5f6ef947cacb64e0f296786bdfb042d1e38113f37ca4edb1ef61cecbde3"}
Oct 30 21:09:29.242975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:29.242954    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:09:29.245080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:29.245052    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z" event={"ID":"297f5eb4-df4e-47dc-bb9b-41505d5da2be","Type":"ContainerStarted","Data":"abfe8b516bbba1aa81460876caede98edefbb6690a5366e03e4cb35aaa80f370"}
Oct 30 21:09:29.245215 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:29.245092    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z" event={"ID":"297f5eb4-df4e-47dc-bb9b-41505d5da2be","Type":"ContainerStarted","Data":"09356e5fd150c45113e9c7ef3515c9856dcc8858f356294479a19dff1ade4189"}
Oct 30 21:09:29.245215 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:29.245127    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:09:29.298882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:29.276212    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk" podStartSLOduration=2.276192948 podStartE2EDuration="2.276192948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:29.263768289 +0000 UTC m=+1935.755584833" watchObservedRunningTime="2026-10-30 21:09:29.276192948 +0000 UTC m=+1935.768009497"
Oct 30 21:09:29.298882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:29.284019    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z" podStartSLOduration=2.283999601 podStartE2EDuration="2.283999601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:29.281356175 +0000 UTC m=+1935.773172720" watchObservedRunningTime="2026-10-30 21:09:29.283999601 +0000 UTC m=+1935.775816148"
Oct 30 21:09:30.341030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.340980    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk" podUID="15a08552-c32c-4113-ad76-d9bfb70d721e" containerName="main" containerID="cri-o://eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e" gracePeriod=45
Oct 30 21:09:30.690597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.688882    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:09:30.849001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.848587    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-tmp-dir\") pod \"15a08552-c32c-4113-ad76-d9bfb70d721e\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") "
Oct 30 21:09:30.849001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.848686    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-model-cache\") pod \"15a08552-c32c-4113-ad76-d9bfb70d721e\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") "
Oct 30 21:09:30.849001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.848758    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-home\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-home\") pod \"15a08552-c32c-4113-ad76-d9bfb70d721e\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") "
Oct 30 21:09:30.849001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.848787    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a08552-c32c-4113-ad76-d9bfb70d721e-kube-api-access-47mzg\" (UniqueName: \"kubernetes.io/projected/15a08552-c32c-4113-ad76-d9bfb70d721e-kube-api-access-47mzg\") pod \"15a08552-c32c-4113-ad76-d9bfb70d721e\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") "
Oct 30 21:09:30.849001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.848965    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-dshm\") pod \"15a08552-c32c-4113-ad76-d9bfb70d721e\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") "
Oct 30 21:09:30.849412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.849014    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a08552-c32c-4113-ad76-d9bfb70d721e-tls-certs\" (UniqueName: \"kubernetes.io/secret/15a08552-c32c-4113-ad76-d9bfb70d721e-tls-certs\") pod \"15a08552-c32c-4113-ad76-d9bfb70d721e\" (UID: \"15a08552-c32c-4113-ad76-d9bfb70d721e\") "
Oct 30 21:09:30.849412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.849032    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-model-cache" pod "15a08552-c32c-4113-ad76-d9bfb70d721e" (UID: "15a08552-c32c-4113-ad76-d9bfb70d721e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:30.849412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.849201    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-tmp-dir" pod "15a08552-c32c-4113-ad76-d9bfb70d721e" (UID: "15a08552-c32c-4113-ad76-d9bfb70d721e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:30.849588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.849540    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-home" pod "15a08552-c32c-4113-ad76-d9bfb70d721e" (UID: "15a08552-c32c-4113-ad76-d9bfb70d721e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:30.850650 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.850181    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:30.850805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.850788    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:30.859055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.859019    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-dshm" pod "15a08552-c32c-4113-ad76-d9bfb70d721e" (UID: "15a08552-c32c-4113-ad76-d9bfb70d721e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:30.859055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.859022    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a08552-c32c-4113-ad76-d9bfb70d721e-tls-certs" pod "15a08552-c32c-4113-ad76-d9bfb70d721e" (UID: "15a08552-c32c-4113-ad76-d9bfb70d721e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:30.860114 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.860082    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a08552-c32c-4113-ad76-d9bfb70d721e-kube-api-access-47mzg" pod "15a08552-c32c-4113-ad76-d9bfb70d721e" (UID: "15a08552-c32c-4113-ad76-d9bfb70d721e"). InnerVolumeSpecName "kube-api-access-47mzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:30.952163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.952130    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:30.952163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.952159    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mzg\" (UniqueName: \"kubernetes.io/projected/15a08552-c32c-4113-ad76-d9bfb70d721e-kube-api-access-47mzg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:30.952163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.952169    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15a08552-c32c-4113-ad76-d9bfb70d721e-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:30.952416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.952179    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15a08552-c32c-4113-ad76-d9bfb70d721e-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:30.992508 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:30.992449    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk" podUID="bc395116-6963-4b0e-b42e-38336983afdf" containerName="payload-processing" containerID="cri-o://f2bab10bdca213f6ab125dfb817a49c2b1a60e534ae166f0307895f549b6b03b" gracePeriod=25
Oct 30 21:09:31.169643 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:09:31.169457    4810 logging.go:55] [core] [Channel #1544 SubChannel #1545]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.229:50052", ServerName: "10.132.0.229:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.229:50052: connect: connection refused"
Oct 30 21:09:31.288719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.286651    4810 generic.go:356] "Generic (PLEG): container finished" podID="15a08552-c32c-4113-ad76-d9bfb70d721e" containerID="eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e" exitCode=0
Oct 30 21:09:31.288719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.286741    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk" event={"ID":"15a08552-c32c-4113-ad76-d9bfb70d721e","Type":"ContainerDied","Data":"eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e"}
Oct 30 21:09:31.288719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.286779    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk" event={"ID":"15a08552-c32c-4113-ad76-d9bfb70d721e","Type":"ContainerDied","Data":"014abd974848dd8ae90bf649a48e8b94fdecc771a98e5ce1e76a449bf4d26b2f"}
Oct 30 21:09:31.288719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.286801    4810 scope.go:122] "RemoveContainer" containerID="eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e"
Oct 30 21:09:31.288719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.286987    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"
Oct 30 21:09:31.303255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.293310    4810 generic.go:356] "Generic (PLEG): container finished" podID="bc395116-6963-4b0e-b42e-38336983afdf" containerID="f2bab10bdca213f6ab125dfb817a49c2b1a60e534ae166f0307895f549b6b03b" exitCode=0
Oct 30 21:09:31.303255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.293552    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk" event={"ID":"bc395116-6963-4b0e-b42e-38336983afdf","Type":"ContainerDied","Data":"f2bab10bdca213f6ab125dfb817a49c2b1a60e534ae166f0307895f549b6b03b"}
Oct 30 21:09:31.314733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.314712    4810 scope.go:122] "RemoveContainer" containerID="eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e"
Oct 30 21:09:31.315660 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:09:31.315599    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e\": container with ID starting with eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e not found: ID does not exist" containerID="eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e"
Oct 30 21:09:31.315660 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.315634    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e"} err="failed to get container status \"eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e\": rpc error: code = NotFound desc = could not find container \"eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e\": container with ID starting with eb4fdf11c013bbf0fc67c2c803e5028afedc42c3d9027236cce9ec2296e3f49e not found: ID does not exist"
Oct 30 21:09:31.380162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.380127    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"]
Oct 30 21:09:31.475336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.474314    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-435414/autores-model-435414-kserve-78cdfcd9f7-64tpk"]
Oct 30 21:09:31.508300 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.508273    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:09:31.595334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.595281    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc395116-6963-4b0e-b42e-38336983afdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc395116-6963-4b0e-b42e-38336983afdf-plugins-config-volume\") pod \"bc395116-6963-4b0e-b42e-38336983afdf\" (UID: \"bc395116-6963-4b0e-b42e-38336983afdf\") "
Oct 30 21:09:31.595575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.595556    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc395116-6963-4b0e-b42e-38336983afdf-kube-api-access-fppft\" (UniqueName: \"kubernetes.io/projected/bc395116-6963-4b0e-b42e-38336983afdf-kube-api-access-fppft\") pod \"bc395116-6963-4b0e-b42e-38336983afdf\" (UID: \"bc395116-6963-4b0e-b42e-38336983afdf\") "
Oct 30 21:09:31.595757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.595734    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc395116-6963-4b0e-b42e-38336983afdf-plugins-config-volume" pod "bc395116-6963-4b0e-b42e-38336983afdf" (UID: "bc395116-6963-4b0e-b42e-38336983afdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:31.596186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.596160    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc395116-6963-4b0e-b42e-38336983afdf-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:31.597883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.597840    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc395116-6963-4b0e-b42e-38336983afdf-kube-api-access-fppft" pod "bc395116-6963-4b0e-b42e-38336983afdf" (UID: "bc395116-6963-4b0e-b42e-38336983afdf"). InnerVolumeSpecName "kube-api-access-fppft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:31.663838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.663798    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a08552-c32c-4113-ad76-d9bfb70d721e" path="/var/lib/kubelet/pods/15a08552-c32c-4113-ad76-d9bfb70d721e/volumes"
Oct 30 21:09:31.697573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:31.697533    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fppft\" (UniqueName: \"kubernetes.io/projected/bc395116-6963-4b0e-b42e-38336983afdf-kube-api-access-fppft\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:32.169201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:32.169156    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk" podUID="bc395116-6963-4b0e-b42e-38336983afdf" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.229:50052\" within 1s: context deadline exceeded"
Oct 30 21:09:32.310879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:32.306133    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"
Oct 30 21:09:32.310879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:32.306133    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk" event={"ID":"bc395116-6963-4b0e-b42e-38336983afdf","Type":"ContainerDied","Data":"ef1d7081dd805f875fe44ac9ca4be3ea1c682379fbb8bb8fe2706d8820a0ee9a"}
Oct 30 21:09:32.310879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:32.306266    4810 scope.go:122] "RemoveContainer" containerID="f2bab10bdca213f6ab125dfb817a49c2b1a60e534ae166f0307895f549b6b03b"
Oct 30 21:09:32.330734 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:32.330696    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"]
Oct 30 21:09:32.336202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:32.336174    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-435414-6f9b5df87-kwdfk"]
Oct 30 21:09:33.664129 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:33.664102    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc395116-6963-4b0e-b42e-38336983afdf" path="/var/lib/kubelet/pods/bc395116-6963-4b0e-b42e-38336983afdf/volumes"
Oct 30 21:09:37.289593 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:37.289553    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"]
Oct 30 21:09:37.293892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:37.293848    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-435414-j86bx"]
Oct 30 21:09:37.663169 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:37.663135    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191256f6-75a7-4f7a-8900-69cb82192599" path="/var/lib/kubelet/pods/191256f6-75a7-4f7a-8900-69cb82192599/volumes"
Oct 30 21:09:40.258221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:40.258188    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:09:40.258681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:40.258265    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:09:41.309516 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.309475    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"]
Oct 30 21:09:41.310505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.310482    4810 cpu_manager.go:414] "containerMap: removing container" podUID="15a08552-c32c-4113-ad76-d9bfb70d721e" containerName="main"
Oct 30 21:09:41.310505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.310506    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a08552-c32c-4113-ad76-d9bfb70d721e" containerName="main"
Oct 30 21:09:41.310624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.310529    4810 cpu_manager.go:414] "containerMap: removing container" podUID="bc395116-6963-4b0e-b42e-38336983afdf" containerName="payload-processing"
Oct 30 21:09:41.310624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.310535    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc395116-6963-4b0e-b42e-38336983afdf" containerName="payload-processing"
Oct 30 21:09:41.310731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.310658    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a08552-c32c-4113-ad76-d9bfb70d721e" containerName="main"
Oct 30 21:09:41.310731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.310671    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc395116-6963-4b0e-b42e-38336983afdf" containerName="payload-processing"
Oct 30 21:09:41.315714 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.315692    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.323647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.323616    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a574c2\"/\"agc-cred-dff81f233dde\""
Oct 30 21:09:41.328565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.328540    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"]
Oct 30 21:09:41.401941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.401909    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.402128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.402071    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52dzd\" (UniqueName: \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-kube-api-access-52dzd\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.402128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.402120    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.402209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.402188    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.503620 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.503580    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.503824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.503694    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52dzd\" (UniqueName: \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-kube-api-access-52dzd\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.503824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.503743    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.503824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.503772    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.504486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.504458    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.504579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.504547    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.506230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.506211    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.513733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.513707    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52dzd\" (UniqueName: \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-kube-api-access-52dzd\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.632429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.632401    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:41.917828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:41.917795    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"]
Oct 30 21:09:41.918932 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:09:41.918898    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a2bdcc_def0_4920_a3a8_4e1c083b96db.slice/crio-59e4615b9dce8a8f5e289f35d19534d02f490b7f09668550b3134582796af6b5 WatchSource:0}: Error finding container 59e4615b9dce8a8f5e289f35d19534d02f490b7f09668550b3134582796af6b5: Status 404 returned error can't find the container with id 59e4615b9dce8a8f5e289f35d19534d02f490b7f09668550b3134582796af6b5
Oct 30 21:09:42.416003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:42.415959    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759" event={"ID":"48a2bdcc-def0-4920-a3a8-4e1c083b96db","Type":"ContainerStarted","Data":"42ffbb9f84411936ad8e18bd0cbc7e2c6e1c814f853151cddd9d3879570839b8"}
Oct 30 21:09:42.416520 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:42.416025    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759" event={"ID":"48a2bdcc-def0-4920-a3a8-4e1c083b96db","Type":"ContainerStarted","Data":"59e4615b9dce8a8f5e289f35d19534d02f490b7f09668550b3134582796af6b5"}
Oct 30 21:09:42.416520 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:42.416065    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:42.441759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:42.441704    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759" podStartSLOduration=1.441690454 podStartE2EDuration="1.441690454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:42.437990448 +0000 UTC m=+1948.929806992" watchObservedRunningTime="2026-10-30 21:09:42.441690454 +0000 UTC m=+1948.933507012"
Oct 30 21:09:44.869918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:44.869885    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"]
Oct 30 21:09:44.870399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:44.870125    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy" containerID="cri-o://d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264" gracePeriod=30
Oct 30 21:09:50.208767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.208735    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 21:09:50.268037 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.268002    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-podinfo\") pod \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") "
Oct 30 21:09:50.268241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.268142    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-certs\") pod \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") "
Oct 30 21:09:50.268241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.268227    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istiod-ca-cert\") pod \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") "
Oct 30 21:09:50.268397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.268261    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-credential-socket\") pod \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") "
Oct 30 21:09:50.268397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.268340    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-envoy\") pod \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") "
Oct 30 21:09:50.268397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.268384    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-socket\") pod \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") "
Oct 30 21:09:50.268622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.268464    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-data\") pod \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") "
Oct 30 21:09:50.268622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.268502    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-token\" (UniqueName: \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-token\") pod \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") "
Oct 30 21:09:50.268622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.268560    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-kube-api-access-2mfzm\" (UniqueName: \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-kube-api-access-2mfzm\") pod \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\" (UID: \"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30\") "
Oct 30 21:09:50.269005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.268978    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-socket" pod "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" (UID: "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:50.269108 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.269085    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-data" pod "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" (UID: "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:50.269302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.269276    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:50.272784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.269310    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:50.272784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.269481    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istiod-ca-cert" pod "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" (UID: "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:50.272784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.269755    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-certs" pod "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" (UID: "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:50.272784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.269911    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-credential-socket" pod "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" (UID: "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:50.272784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.272086    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-token" pod "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" (UID: "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:50.274117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.274090    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-podinfo" pod "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" (UID: "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:09:50.276690 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.276654    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-envoy" pod "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" (UID: "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:50.277041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.277015    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-kube-api-access-2mfzm" pod "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" (UID: "5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30"). InnerVolumeSpecName "kube-api-access-2mfzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:50.370611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.370575    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:50.370611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.370607    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:50.370611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.370617    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:50.370883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.370627    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:50.370883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.370637    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mfzm\" (UniqueName: \"kubernetes.io/projected/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-kube-api-access-2mfzm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:50.370883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.370646    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:50.370883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.370655    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:50.549719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.549640    4810 generic.go:356] "Generic (PLEG): container finished" podID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerID="d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264" exitCode=0
Oct 30 21:09:50.549719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.549676    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" event={"ID":"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30","Type":"ContainerDied","Data":"d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264"}
Oct 30 21:09:50.549719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.549688    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"
Oct 30 21:09:50.549719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.549717    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k" event={"ID":"5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30","Type":"ContainerDied","Data":"c361ac7b5dc1137b873a1bea06c445cae347a8befba25ef223e6c38c6efd45a1"}
Oct 30 21:09:50.550023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.549738    4810 scope.go:122] "RemoveContainer" containerID="d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264"
Oct 30 21:09:50.562832 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.562812    4810 scope.go:122] "RemoveContainer" containerID="d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264"
Oct 30 21:09:50.563180 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:09:50.563156    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264\": container with ID starting with d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264 not found: ID does not exist" containerID="d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264"
Oct 30 21:09:50.563272 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.563186    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264"} err="failed to get container status \"d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264\": rpc error: code = NotFound desc = could not find container \"d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264\": container with ID starting with d4aaab4ede8c0f06f287568281330146e0e0925b3fa99e600420f4b59f0fa264 not found: ID does not exist"
Oct 30 21:09:50.577918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.577877    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"]
Oct 30 21:09:50.584547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:50.584524    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-9755k"]
Oct 30 21:09:51.663525 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:51.663492    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" path="/var/lib/kubelet/pods/5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30/volumes"
Oct 30 21:09:52.592341 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.592295    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"]
Oct 30 21:09:52.593093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.593077    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy"
Oct 30 21:09:52.593150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.593094    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy"
Oct 30 21:09:52.593266 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.593254    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e436e17-6eb0-4c0e-b7b9-ecbbf22c9b30" containerName="istio-proxy"
Oct 30 21:09:52.598606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.598578    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:09:52.603106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.603062    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e8063c\""
Oct 30 21:09:52.604923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.603220    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e8063c-dockercfg-59hl5\""
Oct 30 21:09:52.605959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.605933    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"]
Oct 30 21:09:52.644565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.644531    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"]
Oct 30 21:09:52.649279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.649259    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:09:52.657924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.657897    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"]
Oct 30 21:09:52.706887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.701804    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a8c293f-f822-4033-99cc-0640e9013e20-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx\" (UID: \"7a8c293f-f822-4033-99cc-0640e9013e20\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:09:52.706887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.702047    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf9dn\" (UniqueName: \"kubernetes.io/projected/7a8c293f-f822-4033-99cc-0640e9013e20-kube-api-access-gf9dn\") pod \"payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx\" (UID: \"7a8c293f-f822-4033-99cc-0640e9013e20\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:09:52.706887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.702218    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9dm\" (UniqueName: \"kubernetes.io/projected/967070ba-c779-46ce-8256-cd93fab76c38-kube-api-access-cz9dm\") pod \"payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6\" (UID: \"967070ba-c779-46ce-8256-cd93fab76c38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:09:52.706887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.702272    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967070ba-c779-46ce-8256-cd93fab76c38-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6\" (UID: \"967070ba-c779-46ce-8256-cd93fab76c38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:09:52.809426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.809378    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a8c293f-f822-4033-99cc-0640e9013e20-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx\" (UID: \"7a8c293f-f822-4033-99cc-0640e9013e20\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:09:52.809632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.809519    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf9dn\" (UniqueName: \"kubernetes.io/projected/7a8c293f-f822-4033-99cc-0640e9013e20-kube-api-access-gf9dn\") pod \"payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx\" (UID: \"7a8c293f-f822-4033-99cc-0640e9013e20\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:09:52.809702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.809677    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9dm\" (UniqueName: \"kubernetes.io/projected/967070ba-c779-46ce-8256-cd93fab76c38-kube-api-access-cz9dm\") pod \"payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6\" (UID: \"967070ba-c779-46ce-8256-cd93fab76c38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:09:52.809761 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.809733    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967070ba-c779-46ce-8256-cd93fab76c38-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6\" (UID: \"967070ba-c779-46ce-8256-cd93fab76c38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:09:52.849845 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.846719    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a8c293f-f822-4033-99cc-0640e9013e20-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx\" (UID: \"7a8c293f-f822-4033-99cc-0640e9013e20\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:09:52.852637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.852609    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf9dn\" (UniqueName: \"kubernetes.io/projected/7a8c293f-f822-4033-99cc-0640e9013e20-kube-api-access-gf9dn\") pod \"payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx\" (UID: \"7a8c293f-f822-4033-99cc-0640e9013e20\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:09:52.853746 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.853722    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967070ba-c779-46ce-8256-cd93fab76c38-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6\" (UID: \"967070ba-c779-46ce-8256-cd93fab76c38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:09:52.855847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.855791    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9dm\" (UniqueName: \"kubernetes.io/projected/967070ba-c779-46ce-8256-cd93fab76c38-kube-api-access-cz9dm\") pod \"payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6\" (UID: \"967070ba-c779-46ce-8256-cd93fab76c38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:09:52.918823 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.918784    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:09:52.980796 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:52.980762    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:09:53.299044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:53.297137    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"]
Oct 30 21:09:53.428137 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:53.428112    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:09:53.489652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:53.489616    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"]
Oct 30 21:09:53.512130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:53.512105    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"]
Oct 30 21:09:53.584409 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:53.584377    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx" event={"ID":"7a8c293f-f822-4033-99cc-0640e9013e20","Type":"ContainerStarted","Data":"2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840"}
Oct 30 21:09:53.584562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:53.584439    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx" event={"ID":"7a8c293f-f822-4033-99cc-0640e9013e20","Type":"ContainerStarted","Data":"9dd6779869c0cf05e789a93c9a8eeec043ec0a0ffde5cdac0d65121df3c47422"}
Oct 30 21:09:53.584562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:53.584471    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:09:53.602544 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:53.602465    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6" event={"ID":"967070ba-c779-46ce-8256-cd93fab76c38","Type":"ContainerStarted","Data":"dc4515a47bb02f1f65d4f7b1b4e436de5f5e182c3097d088aa359240b722531e"}
Oct 30 21:09:53.627840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:53.627794    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx" podStartSLOduration=1.627778768 podStartE2EDuration="1.627778768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:53.624109461 +0000 UTC m=+1960.115926007" watchObservedRunningTime="2026-10-30 21:09:53.627778768 +0000 UTC m=+1960.119595316"
Oct 30 21:09:54.614428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:54.614391    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6" event={"ID":"967070ba-c779-46ce-8256-cd93fab76c38","Type":"ContainerStarted","Data":"8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2"}
Oct 30 21:09:54.637220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:54.637174    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6" podStartSLOduration=2.637161209 podStartE2EDuration="2.637161209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:54.633147765 +0000 UTC m=+1961.124964311" watchObservedRunningTime="2026-10-30 21:09:54.637161209 +0000 UTC m=+1961.128977753"
Oct 30 21:09:55.157806 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:09:55.157765    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:09:55.623792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:55.623704    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:09:57.748132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.748095    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"]
Oct 30 21:09:57.754218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.754194    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.764312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.764268    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7386b4\"/\"explicit-model-7386b4-kserve-self-signed-certs\""
Oct 30 21:09:57.765440 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.765414    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7386b4\"/\"openshift-service-ca.crt\""
Oct 30 21:09:57.765616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.765490    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7386b4\"/\"default-dockercfg-6sbng\""
Oct 30 21:09:57.765709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.765603    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7386b4\"/\"kube-root-ca.crt\""
Oct 30 21:09:57.770326 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.770299    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"]
Oct 30 21:09:57.778972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.778901    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-model-cache\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.779219 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.779199    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-home\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.779478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.779454    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-dshm\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.779669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.779598    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tls-certs\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.779933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.779908    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tmp-dir\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.780028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.779977    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cvcd\" (UniqueName: \"kubernetes.io/projected/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-kube-api-access-4cvcd\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.881597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.881555    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-home\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.881969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.881942    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-dshm\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.882080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.881966    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-home\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.882080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.882001    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tls-certs\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.882080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.882064    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tmp-dir\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.882255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.882127    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cvcd\" (UniqueName: \"kubernetes.io/projected/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-kube-api-access-4cvcd\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.882255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.882220    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-model-cache\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.882619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.882592    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-model-cache\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.882929 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.882902    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tmp-dir\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.884825 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.884800    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-dshm\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.885028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.885011    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tls-certs\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:57.891032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:57.891007    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cvcd\" (UniqueName: \"kubernetes.io/projected/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-kube-api-access-4cvcd\") pod \"explicit-model-7386b4-kserve-57b89f89c5-69dwn\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") " pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:58.094952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.094091    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:58.442820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.442776    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"]
Oct 30 21:09:58.490608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.490576    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv" podUID="369f4f0c-b013-4f17-b40e-6c15e995337f" containerName="payload-processing" containerID="cri-o://03849b9ec858ba3b61455ee0958e68138efbf66d6102850dec0cbde895a7068a" gracePeriod=25
Oct 30 21:09:58.654613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.654578    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn" event={"ID":"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39","Type":"ContainerStarted","Data":"0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b"}
Oct 30 21:09:58.654613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.654627    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn" event={"ID":"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39","Type":"ContainerStarted","Data":"14bd4e5d93d63a2f35605d5589976868e51259be3e113fb0fc55b678eae150fe"}
Oct 30 21:09:58.657218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.657188    4810 generic.go:356] "Generic (PLEG): container finished" podID="369f4f0c-b013-4f17-b40e-6c15e995337f" containerID="03849b9ec858ba3b61455ee0958e68138efbf66d6102850dec0cbde895a7068a" exitCode=0
Oct 30 21:09:58.657408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.657381    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv" event={"ID":"369f4f0c-b013-4f17-b40e-6c15e995337f","Type":"ContainerDied","Data":"03849b9ec858ba3b61455ee0958e68138efbf66d6102850dec0cbde895a7068a"}
Oct 30 21:09:58.680586 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.680525    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn" podStartSLOduration=1.680505543 podStartE2EDuration="1.680505543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:58.671061854 +0000 UTC m=+1965.162878401" watchObservedRunningTime="2026-10-30 21:09:58.680505543 +0000 UTC m=+1965.172322110"
Oct 30 21:09:58.764693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.764663    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:09:58.827716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.827671    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-routing-overlay-volume\") pod \"369f4f0c-b013-4f17-b40e-6c15e995337f\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") "
Oct 30 21:09:58.828013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.827756    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-provider-credentials\") pod \"369f4f0c-b013-4f17-b40e-6c15e995337f\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") "
Oct 30 21:09:58.828013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.827936    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-kube-api-access-xjz6f\" (UniqueName: \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-kube-api-access-xjz6f\") pod \"369f4f0c-b013-4f17-b40e-6c15e995337f\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") "
Oct 30 21:09:58.828193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.828173    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-plugins-config-volume\") pod \"369f4f0c-b013-4f17-b40e-6c15e995337f\" (UID: \"369f4f0c-b013-4f17-b40e-6c15e995337f\") "
Oct 30 21:09:58.828348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.828317    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-routing-overlay-volume" pod "369f4f0c-b013-4f17-b40e-6c15e995337f" (UID: "369f4f0c-b013-4f17-b40e-6c15e995337f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:58.828767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.828745    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:58.829176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.829150    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-plugins-config-volume" pod "369f4f0c-b013-4f17-b40e-6c15e995337f" (UID: "369f4f0c-b013-4f17-b40e-6c15e995337f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:58.836527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.836497    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-provider-credentials" pod "369f4f0c-b013-4f17-b40e-6c15e995337f" (UID: "369f4f0c-b013-4f17-b40e-6c15e995337f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:58.838739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.838711    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-kube-api-access-xjz6f" pod "369f4f0c-b013-4f17-b40e-6c15e995337f" (UID: "369f4f0c-b013-4f17-b40e-6c15e995337f"). InnerVolumeSpecName "kube-api-access-xjz6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:58.930768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.930090    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjz6f\" (UniqueName: \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-kube-api-access-xjz6f\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:58.930768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.930129    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369f4f0c-b013-4f17-b40e-6c15e995337f-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:58.930768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:58.930145    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/369f4f0c-b013-4f17-b40e-6c15e995337f-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:59.095163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:59.095085    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:09:59.667208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:59.667173    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"
Oct 30 21:09:59.667419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:59.667369    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv" event={"ID":"369f4f0c-b013-4f17-b40e-6c15e995337f","Type":"ContainerDied","Data":"096c73066bb6204acd7e39d04de6ff1a5b3f5cd7888291eb7060584919502391"}
Oct 30 21:09:59.667497 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:59.667420    4810 scope.go:122] "RemoveContainer" containerID="03849b9ec858ba3b61455ee0958e68138efbf66d6102850dec0cbde895a7068a"
Oct 30 21:09:59.701665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:59.701633    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"]
Oct 30 21:09:59.705285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:09:59.705264    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cb5mcv"]
Oct 30 21:10:01.663652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.663621    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369f4f0c-b013-4f17-b40e-6c15e995337f" path="/var/lib/kubelet/pods/369f4f0c-b013-4f17-b40e-6c15e995337f/volumes"
Oct 30 21:10:01.900836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.900784    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"]
Oct 30 21:10:01.901701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.901678    4810 cpu_manager.go:414] "containerMap: removing container" podUID="369f4f0c-b013-4f17-b40e-6c15e995337f" containerName="payload-processing"
Oct 30 21:10:01.901701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.901701    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369f4f0c-b013-4f17-b40e-6c15e995337f" containerName="payload-processing"
Oct 30 21:10:01.901922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.901902    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="369f4f0c-b013-4f17-b40e-6c15e995337f" containerName="payload-processing"
Oct 30 21:10:01.907353 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.907329    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:01.919168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.919088    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"]
Oct 30 21:10:01.919316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.919293    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a574c2\"/\"agc-cred-1373999ee180\""
Oct 30 21:10:01.978039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.977987    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:01.978223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.978179    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:01.978277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.978236    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5s7\" (UniqueName: \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-kube-api-access-tb5s7\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:01.978362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:01.978347    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.079259 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.079160    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.079467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.079365    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.079467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.079436    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5s7\" (UniqueName: \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-kube-api-access-tb5s7\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.079601 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.079546    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.080362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.080330    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.080477 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.080335    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.082797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.082764    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.089969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.089944    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5s7\" (UniqueName: \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-kube-api-access-tb5s7\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.231833 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.231718    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.539420 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:10:02.539370    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb31954e_6cf1_4bc4_b878_23dda9ca4b0c.slice/crio-490bbabd8324f3dcd4c545bc682016f8344181780d05ac12342ebf9b6476b6a3 WatchSource:0}: Error finding container 490bbabd8324f3dcd4c545bc682016f8344181780d05ac12342ebf9b6476b6a3: Status 404 returned error can't find the container with id 490bbabd8324f3dcd4c545bc682016f8344181780d05ac12342ebf9b6476b6a3
Oct 30 21:10:02.539686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.539655    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"]
Oct 30 21:10:02.711277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.711225    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g" event={"ID":"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c","Type":"ContainerStarted","Data":"8768f92d665dd10d7ef01049b8cfd4143466c223c4479b557782cb4a5b24d566"}
Oct 30 21:10:02.711939 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.711290    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g" event={"ID":"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c","Type":"ContainerStarted","Data":"490bbabd8324f3dcd4c545bc682016f8344181780d05ac12342ebf9b6476b6a3"}
Oct 30 21:10:02.711939 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.711321    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:02.731372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:02.731325    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g" podStartSLOduration=1.7313109309999999 podStartE2EDuration="1.731310931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:02.729015281 +0000 UTC m=+1969.220831827" watchObservedRunningTime="2026-10-30 21:10:02.731310931 +0000 UTC m=+1969.223127476"
Oct 30 21:10:04.619463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:04.619433    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:10:05.569904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.569851    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"]
Oct 30 21:10:05.850012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.849929    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"]
Oct 30 21:10:05.864839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.864803    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"]
Oct 30 21:10:05.865026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.864936    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:05.870616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.870591    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:10:05.870781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.870717    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:10:05.880926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.880887    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95c94\" (UniqueName: \"kubernetes.io/projected/a33dc572-9850-4578-be7d-a4cea9daa62c-kube-api-access-95c94\") pod \"maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:05.881239 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.881220    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:05.881421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.881396    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:05.982253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.982220    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95c94\" (UniqueName: \"kubernetes.io/projected/a33dc572-9850-4578-be7d-a4cea9daa62c-kube-api-access-95c94\") pod \"maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:05.982412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.982319    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:05.982412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.982343    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:05.983220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.983185    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:05.985099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.985070    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:05.994399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:05.994367    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95c94\" (UniqueName: \"kubernetes.io/projected/a33dc572-9850-4578-be7d-a4cea9daa62c-kube-api-access-95c94\") pod \"maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:06.199418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:06.199382    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:06.483141 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:10:06.483093    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33dc572_9850_4578_be7d_a4cea9daa62c.slice/crio-14c9e192dc927a2b427d8bfa8df0f19a7be9debfa67c8d271c6b792d562ef129 WatchSource:0}: Error finding container 14c9e192dc927a2b427d8bfa8df0f19a7be9debfa67c8d271c6b792d562ef129: Status 404 returned error can't find the container with id 14c9e192dc927a2b427d8bfa8df0f19a7be9debfa67c8d271c6b792d562ef129
Oct 30 21:10:06.484269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:06.484131    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"]
Oct 30 21:10:06.637906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:06.637841    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:10:06.825803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:06.825713    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4" event={"ID":"a33dc572-9850-4578-be7d-a4cea9daa62c","Type":"ContainerStarted","Data":"14c9e192dc927a2b427d8bfa8df0f19a7be9debfa67c8d271c6b792d562ef129"}
Oct 30 21:10:07.836563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:07.836474    4810 generic.go:356] "Generic (PLEG): container finished" podID="a33dc572-9850-4578-be7d-a4cea9daa62c" containerID="336c48ab0ce2146ffa9fbd90e6c5583e1d8866d5b62d7abd853bf3f129400f6d" exitCode=0
Oct 30 21:10:07.836967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:07.836568    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4" event={"ID":"a33dc572-9850-4578-be7d-a4cea9daa62c","Type":"ContainerDied","Data":"336c48ab0ce2146ffa9fbd90e6c5583e1d8866d5b62d7abd853bf3f129400f6d"}
Oct 30 21:10:09.030924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.030886    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:09.137334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.137303    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33dc572-9850-4578-be7d-a4cea9daa62c-kube-api-access-95c94\" (UniqueName: \"kubernetes.io/projected/a33dc572-9850-4578-be7d-a4cea9daa62c-kube-api-access-95c94\") pod \"a33dc572-9850-4578-be7d-a4cea9daa62c\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") "
Oct 30 21:10:09.137507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.137341    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-xks-ca\") pod \"a33dc572-9850-4578-be7d-a4cea9daa62c\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") "
Oct 30 21:10:09.137578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.137564    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-openshift-ca\") pod \"a33dc572-9850-4578-be7d-a4cea9daa62c\" (UID: \"a33dc572-9850-4578-be7d-a4cea9daa62c\") "
Oct 30 21:10:09.137933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.137897    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-openshift-ca" pod "a33dc572-9850-4578-be7d-a4cea9daa62c" (UID: "a33dc572-9850-4578-be7d-a4cea9daa62c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:09.138137 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.138117    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:09.139435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.139392    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33dc572-9850-4578-be7d-a4cea9daa62c-kube-api-access-95c94" pod "a33dc572-9850-4578-be7d-a4cea9daa62c" (UID: "a33dc572-9850-4578-be7d-a4cea9daa62c"). InnerVolumeSpecName "kube-api-access-95c94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:09.139601 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.139583    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-xks-ca" pod "a33dc572-9850-4578-be7d-a4cea9daa62c" (UID: "a33dc572-9850-4578-be7d-a4cea9daa62c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:09.240340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.240294    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95c94\" (UniqueName: \"kubernetes.io/projected/a33dc572-9850-4578-be7d-a4cea9daa62c-kube-api-access-95c94\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:09.240340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.240331    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a33dc572-9850-4578-be7d-a4cea9daa62c-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:09.853620 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.853589    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"
Oct 30 21:10:09.853814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.853618    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4" event={"ID":"a33dc572-9850-4578-be7d-a4cea9daa62c","Type":"ContainerDied","Data":"14c9e192dc927a2b427d8bfa8df0f19a7be9debfa67c8d271c6b792d562ef129"}
Oct 30 21:10:09.853814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:09.853666    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14c9e192dc927a2b427d8bfa8df0f19a7be9debfa67c8d271c6b792d562ef129"
Oct 30 21:10:13.725484 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:13.725450    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:10:13.782422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:13.782382    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"]
Oct 30 21:10:15.401436 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.401404    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"]
Oct 30 21:10:15.402181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.402164    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a33dc572-9850-4578-be7d-a4cea9daa62c" containerName="revoke-keys"
Oct 30 21:10:15.402181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.402180    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33dc572-9850-4578-be7d-a4cea9daa62c" containerName="revoke-keys"
Oct 30 21:10:15.402304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.402281    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33dc572-9850-4578-be7d-a4cea9daa62c" containerName="revoke-keys"
Oct 30 21:10:15.407049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.407028    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.424925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.424894    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"openshift-service-ca.crt\""
Oct 30 21:10:15.425132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.425084    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"agc-cred-7840388e05a6\""
Oct 30 21:10:15.425220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.425127    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"kube-root-ca.crt\""
Oct 30 21:10:15.425220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.425086    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"routing-overlay\""
Oct 30 21:10:15.425220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.425206    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d95261\""
Oct 30 21:10:15.432006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.431973    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"]
Oct 30 21:10:15.574919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.574883    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.575189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.575161    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.575490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.575468    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8qp\" (UniqueName: \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-kube-api-access-6z8qp\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.575696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.575678    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.677461 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.677367    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8qp\" (UniqueName: \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-kube-api-access-6z8qp\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.677461 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.677453    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.677823 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.677795    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.677898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.677846    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.678771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.678745    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.678915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.678758    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.680345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.680321    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.692330 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.692297    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8qp\" (UniqueName: \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-kube-api-access-6z8qp\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:15.742570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:15.742529    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:16.039114 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:10:16.039065    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebfea03_bc7c_4de7_b677_220520b6ab6e.slice/crio-f0e757bebb2ae726e2b3e19e521c4235587efe8bf89f0cc73d785b220cee23c2 WatchSource:0}: Error finding container f0e757bebb2ae726e2b3e19e521c4235587efe8bf89f0cc73d785b220cee23c2: Status 404 returned error can't find the container with id f0e757bebb2ae726e2b3e19e521c4235587efe8bf89f0cc73d785b220cee23c2
Oct 30 21:10:16.049670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:16.049626    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"]
Oct 30 21:10:16.391720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:16.391568    4810 scope.go:122] "RemoveContainer" containerID="7d7c2f36f367f3bc9448aae79030ddfc91a43d6490b4715a1c962d4657ea1900"
Oct 30 21:10:16.922870 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:16.922820    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw" event={"ID":"1ebfea03-bc7c-4de7-b677-220520b6ab6e","Type":"ContainerStarted","Data":"4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6"}
Oct 30 21:10:16.923294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:16.922906    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw" event={"ID":"1ebfea03-bc7c-4de7-b677-220520b6ab6e","Type":"ContainerStarted","Data":"f0e757bebb2ae726e2b3e19e521c4235587efe8bf89f0cc73d785b220cee23c2"}
Oct 30 21:10:16.923294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:16.922939    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:16.942719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:16.942667    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw" podStartSLOduration=1.942654631 podStartE2EDuration="1.942654631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:16.939547968 +0000 UTC m=+1983.431364512" watchObservedRunningTime="2026-10-30 21:10:16.942654631 +0000 UTC m=+1983.434471176"
Oct 30 21:10:18.783445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:18.783396    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759" podUID="48a2bdcc-def0-4920-a3a8-4e1c083b96db" containerName="payload-processing" containerID="cri-o://42ffbb9f84411936ad8e18bd0cbc7e2c6e1c814f853151cddd9d3879570839b8" gracePeriod=25
Oct 30 21:10:18.940874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:18.940821    4810 generic.go:356] "Generic (PLEG): container finished" podID="48a2bdcc-def0-4920-a3a8-4e1c083b96db" containerID="42ffbb9f84411936ad8e18bd0cbc7e2c6e1c814f853151cddd9d3879570839b8" exitCode=0
Oct 30 21:10:18.941065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:18.940909    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759" event={"ID":"48a2bdcc-def0-4920-a3a8-4e1c083b96db","Type":"ContainerDied","Data":"42ffbb9f84411936ad8e18bd0cbc7e2c6e1c814f853151cddd9d3879570839b8"}
Oct 30 21:10:19.092419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.092393    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:10:19.145072 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.145041    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-plugins-config-volume\") pod \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") "
Oct 30 21:10:19.145256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.145129    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-provider-credentials\" (UniqueName: \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-provider-credentials\") pod \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") "
Oct 30 21:10:19.145256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.145243    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-kube-api-access-52dzd\" (UniqueName: \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-kube-api-access-52dzd\") pod \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") "
Oct 30 21:10:19.145437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.145408    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-routing-overlay-volume\") pod \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\" (UID: \"48a2bdcc-def0-4920-a3a8-4e1c083b96db\") "
Oct 30 21:10:19.145568 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.145403    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-plugins-config-volume" pod "48a2bdcc-def0-4920-a3a8-4e1c083b96db" (UID: "48a2bdcc-def0-4920-a3a8-4e1c083b96db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:19.145776 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.145754    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-routing-overlay-volume" pod "48a2bdcc-def0-4920-a3a8-4e1c083b96db" (UID: "48a2bdcc-def0-4920-a3a8-4e1c083b96db"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:19.146005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.145987    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:19.146005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.146009    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a2bdcc-def0-4920-a3a8-4e1c083b96db-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:19.147354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.147331    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-kube-api-access-52dzd" pod "48a2bdcc-def0-4920-a3a8-4e1c083b96db" (UID: "48a2bdcc-def0-4920-a3a8-4e1c083b96db"). InnerVolumeSpecName "kube-api-access-52dzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:19.147354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.147348    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-provider-credentials" pod "48a2bdcc-def0-4920-a3a8-4e1c083b96db" (UID: "48a2bdcc-def0-4920-a3a8-4e1c083b96db"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:19.247267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.247234    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:19.247267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.247261    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52dzd\" (UniqueName: \"kubernetes.io/projected/48a2bdcc-def0-4920-a3a8-4e1c083b96db-kube-api-access-52dzd\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:19.950176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.950093    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"
Oct 30 21:10:19.950176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.950122    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759" event={"ID":"48a2bdcc-def0-4920-a3a8-4e1c083b96db","Type":"ContainerDied","Data":"59e4615b9dce8a8f5e289f35d19534d02f490b7f09668550b3134582796af6b5"}
Oct 30 21:10:19.950700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.950186    4810 scope.go:122] "RemoveContainer" containerID="42ffbb9f84411936ad8e18bd0cbc7e2c6e1c814f853151cddd9d3879570839b8"
Oct 30 21:10:19.970589 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.970560    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"]
Oct 30 21:10:19.975622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:19.975598    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868qr759"]
Oct 30 21:10:20.604179 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.604113    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn" podUID="3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" containerName="main" containerID="cri-o://0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b" gracePeriod=45
Oct 30 21:10:20.884117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.884089    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:10:20.965820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.965786    4810 generic.go:356] "Generic (PLEG): container finished" podID="3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" containerID="0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b" exitCode=0
Oct 30 21:10:20.966522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.965843    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"
Oct 30 21:10:20.966522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.966013    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn" event={"ID":"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39","Type":"ContainerDied","Data":"0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b"}
Oct 30 21:10:20.966522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.966041    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn" event={"ID":"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39","Type":"ContainerDied","Data":"14bd4e5d93d63a2f35605d5589976868e51259be3e113fb0fc55b678eae150fe"}
Oct 30 21:10:20.966522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.966057    4810 scope.go:122] "RemoveContainer" containerID="0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b"
Oct 30 21:10:20.970843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.970821    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tls-certs\") pod \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") "
Oct 30 21:10:20.970984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.970868    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-model-cache\") pod \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") "
Oct 30 21:10:20.970984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.970948    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-dshm\") pod \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") "
Oct 30 21:10:20.971077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.971032    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-home\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-home\") pod \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") "
Oct 30 21:10:20.971077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.971066    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-kube-api-access-4cvcd\" (UniqueName: \"kubernetes.io/projected/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-kube-api-access-4cvcd\") pod \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") "
Oct 30 21:10:20.971194 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.971095    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tmp-dir\") pod \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\" (UID: \"3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39\") "
Oct 30 21:10:20.971194 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.971179    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-model-cache" pod "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" (UID: "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:20.975739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.971384    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-home" pod "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" (UID: "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:20.975739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.971782    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:20.975739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.971802    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:20.975739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.972021    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tmp-dir" pod "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" (UID: "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:20.975739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.974603    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-kube-api-access-4cvcd" pod "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" (UID: "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39"). InnerVolumeSpecName "kube-api-access-4cvcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:20.975739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.974707    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tls-certs" pod "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" (UID: "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:20.976324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.976298    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-dshm" pod "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" (UID: "3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:20.981741 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.981720    4810 scope.go:122] "RemoveContainer" containerID="0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b"
Oct 30 21:10:20.982083 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:10:20.982056    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b\": container with ID starting with 0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b not found: ID does not exist" containerID="0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b"
Oct 30 21:10:20.982201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:20.982092    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b"} err="failed to get container status \"0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b\": rpc error: code = NotFound desc = could not find container \"0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b\": container with ID starting with 0b4d7e9509ae2970cc693034fc1570caca364061543fd145ac165ca495e3814b not found: ID does not exist"
Oct 30 21:10:21.081750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:21.081176    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cvcd\" (UniqueName: \"kubernetes.io/projected/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-kube-api-access-4cvcd\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:21.081750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:21.081232    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:21.081750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:21.081247    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:21.081750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:21.081260    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:21.294421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:21.294151    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"]
Oct 30 21:10:21.298316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:21.298287    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-7386b4/explicit-model-7386b4-kserve-57b89f89c5-69dwn"]
Oct 30 21:10:21.664489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:21.664457    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" path="/var/lib/kubelet/pods/3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39/volumes"
Oct 30 21:10:21.665107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:21.665085    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a2bdcc-def0-4920-a3a8-4e1c083b96db" path="/var/lib/kubelet/pods/48a2bdcc-def0-4920-a3a8-4e1c083b96db/volumes"
Oct 30 21:10:25.157101 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:10:25.157062    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:10:27.065111 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:27.065075    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:10:27.945135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:27.942576    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:10:29.762945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.762913    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"]
Oct 30 21:10:29.763661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.763639    4810 cpu_manager.go:414] "containerMap: removing container" podUID="48a2bdcc-def0-4920-a3a8-4e1c083b96db" containerName="payload-processing"
Oct 30 21:10:29.763661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.763656    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a2bdcc-def0-4920-a3a8-4e1c083b96db" containerName="payload-processing"
Oct 30 21:10:29.763661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.763667    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" containerName="main"
Oct 30 21:10:29.763977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.763672    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" containerName="main"
Oct 30 21:10:29.763977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.763830    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a2bdcc-def0-4920-a3a8-4e1c083b96db" containerName="payload-processing"
Oct 30 21:10:29.763977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.763842    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae50ccb-44fb-4d9f-9d0c-ee3731ed4f39" containerName="main"
Oct 30 21:10:29.769158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.769140    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:29.774147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.774102    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:10:29.774147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.774102    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:10:29.775198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.775173    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"]
Oct 30 21:10:29.917404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.917372    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141468aa-l5ns2\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:29.917616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.917408    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141468aa-l5ns2\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:29.917616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:29.917498    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56d8c\" (UniqueName: \"kubernetes.io/projected/c869dc88-791f-437c-9b0f-70f46308500c-kube-api-access-56d8c\") pod \"maas-api-revoke-sub-141468aa-l5ns2\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:30.021572 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:30.021469    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141468aa-l5ns2\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:30.021919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:30.021825    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141468aa-l5ns2\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:30.022067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:30.021955    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56d8c\" (UniqueName: \"kubernetes.io/projected/c869dc88-791f-437c-9b0f-70f46308500c-kube-api-access-56d8c\") pod \"maas-api-revoke-sub-141468aa-l5ns2\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:30.022474 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:30.022452    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141468aa-l5ns2\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:30.024059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:30.024036    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141468aa-l5ns2\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:30.031515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:30.031482    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56d8c\" (UniqueName: \"kubernetes.io/projected/c869dc88-791f-437c-9b0f-70f46308500c-kube-api-access-56d8c\") pod \"maas-api-revoke-sub-141468aa-l5ns2\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:30.086974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:30.086943    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:30.376093 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:10:30.376060    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc869dc88_791f_437c_9b0f_70f46308500c.slice/crio-73b4c84ae4b9a41b9e546b0dc455916830e3753110e7779867cb92f2abca5e77 WatchSource:0}: Error finding container 73b4c84ae4b9a41b9e546b0dc455916830e3753110e7779867cb92f2abca5e77: Status 404 returned error can't find the container with id 73b4c84ae4b9a41b9e546b0dc455916830e3753110e7779867cb92f2abca5e77
Oct 30 21:10:30.377722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:30.377261    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"]
Oct 30 21:10:31.072681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:31.072596    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2" event={"ID":"c869dc88-791f-437c-9b0f-70f46308500c","Type":"ContainerStarted","Data":"6181b29cf7acbcb4be7b32818baa582521d8127a9b785eb14e36ca5265b04a73"}
Oct 30 21:10:31.072681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:31.072658    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2" event={"ID":"c869dc88-791f-437c-9b0f-70f46308500c","Type":"ContainerStarted","Data":"73b4c84ae4b9a41b9e546b0dc455916830e3753110e7779867cb92f2abca5e77"}
Oct 30 21:10:31.093645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:31.093597    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2" podStartSLOduration=2.09358313 podStartE2EDuration="2.09358313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:31.088647006 +0000 UTC m=+1997.580463550" watchObservedRunningTime="2026-10-30 21:10:31.09358313 +0000 UTC m=+1997.585399674"
Oct 30 21:10:32.082991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:32.082953    4810 generic.go:356] "Generic (PLEG): container finished" podID="c869dc88-791f-437c-9b0f-70f46308500c" containerID="6181b29cf7acbcb4be7b32818baa582521d8127a9b785eb14e36ca5265b04a73" exitCode=0
Oct 30 21:10:32.083462 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:32.083042    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2" event={"ID":"c869dc88-791f-437c-9b0f-70f46308500c","Type":"ContainerDied","Data":"6181b29cf7acbcb4be7b32818baa582521d8127a9b785eb14e36ca5265b04a73"}
Oct 30 21:10:33.266950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:33.266918    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:33.272357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:33.272317    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-xks-ca\") pod \"c869dc88-791f-437c-9b0f-70f46308500c\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") "
Oct 30 21:10:33.272479 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:33.272403    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c869dc88-791f-437c-9b0f-70f46308500c-kube-api-access-56d8c\" (UniqueName: \"kubernetes.io/projected/c869dc88-791f-437c-9b0f-70f46308500c-kube-api-access-56d8c\") pod \"c869dc88-791f-437c-9b0f-70f46308500c\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") "
Oct 30 21:10:33.272527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:33.272484    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-openshift-ca\") pod \"c869dc88-791f-437c-9b0f-70f46308500c\" (UID: \"c869dc88-791f-437c-9b0f-70f46308500c\") "
Oct 30 21:10:33.273002 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:33.272964    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-openshift-ca" pod "c869dc88-791f-437c-9b0f-70f46308500c" (UID: "c869dc88-791f-437c-9b0f-70f46308500c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:33.274674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:33.274649    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-xks-ca" pod "c869dc88-791f-437c-9b0f-70f46308500c" (UID: "c869dc88-791f-437c-9b0f-70f46308500c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:33.275171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:33.275152    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c869dc88-791f-437c-9b0f-70f46308500c-kube-api-access-56d8c" pod "c869dc88-791f-437c-9b0f-70f46308500c" (UID: "c869dc88-791f-437c-9b0f-70f46308500c"). InnerVolumeSpecName "kube-api-access-56d8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:33.374775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:33.374655    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:33.374775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:33.374693    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56d8c\" (UniqueName: \"kubernetes.io/projected/c869dc88-791f-437c-9b0f-70f46308500c-kube-api-access-56d8c\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:33.374775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:33.374709    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c869dc88-791f-437c-9b0f-70f46308500c-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:34.062066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:34.061976    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"]
Oct 30 21:10:34.065827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:34.065795    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7386b4-fhxl4"]
Oct 30 21:10:34.102585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:34.102555    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"
Oct 30 21:10:34.102585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:34.102572    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2" event={"ID":"c869dc88-791f-437c-9b0f-70f46308500c","Type":"ContainerDied","Data":"73b4c84ae4b9a41b9e546b0dc455916830e3753110e7779867cb92f2abca5e77"}
Oct 30 21:10:34.102763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:34.102617    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b4c84ae4b9a41b9e546b0dc455916830e3753110e7779867cb92f2abca5e77"
Oct 30 21:10:35.663942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:35.663908    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33dc572-9850-4578-be7d-a4cea9daa62c" path="/var/lib/kubelet/pods/a33dc572-9850-4578-be7d-a4cea9daa62c/volumes"
Oct 30 21:10:43.858790 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.858753    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"]
Oct 30 21:10:43.859594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.859577    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c869dc88-791f-437c-9b0f-70f46308500c" containerName="revoke-keys"
Oct 30 21:10:43.859643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.859595    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c869dc88-791f-437c-9b0f-70f46308500c" containerName="revoke-keys"
Oct 30 21:10:43.859757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.859747    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c869dc88-791f-437c-9b0f-70f46308500c" containerName="revoke-keys"
Oct 30 21:10:43.863585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.863568    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:10:43.869350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.869303    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e1e0f5\""
Oct 30 21:10:43.869879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.869420    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e1e0f5-dockercfg-mglqk\""
Oct 30 21:10:43.875602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.875561    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"]
Oct 30 21:10:43.903713 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.903676    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"]
Oct 30 21:10:43.908684 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.908664    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:10:43.919756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.919719    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"]
Oct 30 21:10:43.996132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.996074    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6n9b\" (UniqueName: \"kubernetes.io/projected/19991362-bc52-4564-b54e-64f087a20820-kube-api-access-c6n9b\") pod \"payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c\" (UID: \"19991362-bc52-4564-b54e-64f087a20820\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:10:43.996553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.996519    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghxw\" (UniqueName: \"kubernetes.io/projected/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-kube-api-access-fghxw\") pod \"payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7\" (UID: \"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:10:43.996727 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.996708    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19991362-bc52-4564-b54e-64f087a20820-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c\" (UID: \"19991362-bc52-4564-b54e-64f087a20820\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:10:43.997119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:43.997093    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7\" (UID: \"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:10:44.104920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.103058    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7\" (UID: \"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:10:44.105206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.104772    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7\" (UID: \"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:10:44.105370 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.105215    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6n9b\" (UniqueName: \"kubernetes.io/projected/19991362-bc52-4564-b54e-64f087a20820-kube-api-access-c6n9b\") pod \"payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c\" (UID: \"19991362-bc52-4564-b54e-64f087a20820\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:10:44.105633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.105606    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghxw\" (UniqueName: \"kubernetes.io/projected/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-kube-api-access-fghxw\") pod \"payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7\" (UID: \"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:10:44.105806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.105791    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19991362-bc52-4564-b54e-64f087a20820-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c\" (UID: \"19991362-bc52-4564-b54e-64f087a20820\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:10:44.108946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.106974    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19991362-bc52-4564-b54e-64f087a20820-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c\" (UID: \"19991362-bc52-4564-b54e-64f087a20820\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:10:44.116846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.116812    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6n9b\" (UniqueName: \"kubernetes.io/projected/19991362-bc52-4564-b54e-64f087a20820-kube-api-access-c6n9b\") pod \"payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c\" (UID: \"19991362-bc52-4564-b54e-64f087a20820\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:10:44.121697 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.121671    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghxw\" (UniqueName: \"kubernetes.io/projected/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-kube-api-access-fghxw\") pod \"payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7\" (UID: \"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:10:44.179797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.179751    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:10:44.234327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.234113    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:10:44.490132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.489528    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"]
Oct 30 21:10:44.490290 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:10:44.490198    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366d6f28_7835_4ea2_9ebf_32bfa47f3e0e.slice/crio-49155a591aab653276fb0f84869e1ba5626b381d526649dfae1a2808330dbd6e WatchSource:0}: Error finding container 49155a591aab653276fb0f84869e1ba5626b381d526649dfae1a2808330dbd6e: Status 404 returned error can't find the container with id 49155a591aab653276fb0f84869e1ba5626b381d526649dfae1a2808330dbd6e
Oct 30 21:10:44.614879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:44.612020    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"]
Oct 30 21:10:45.203394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:45.203356    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c" event={"ID":"19991362-bc52-4564-b54e-64f087a20820","Type":"ContainerStarted","Data":"b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6"}
Oct 30 21:10:45.203891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:45.203418    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c" event={"ID":"19991362-bc52-4564-b54e-64f087a20820","Type":"ContainerStarted","Data":"ddd21b5749100932d7097518f71581c1bc9bc03b1fa141dc244a534d96785157"}
Oct 30 21:10:45.203891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:45.203447    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:10:45.205489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:45.205458    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7" event={"ID":"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e","Type":"ContainerStarted","Data":"86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7"}
Oct 30 21:10:45.205603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:45.205507    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7" event={"ID":"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e","Type":"ContainerStarted","Data":"49155a591aab653276fb0f84869e1ba5626b381d526649dfae1a2808330dbd6e"}
Oct 30 21:10:45.205603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:45.205541    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:10:45.223383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:45.223313    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c" podStartSLOduration=2.223295599 podStartE2EDuration="2.223295599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:45.219900373 +0000 UTC m=+2011.711716919" watchObservedRunningTime="2026-10-30 21:10:45.223295599 +0000 UTC m=+2011.715112149"
Oct 30 21:10:45.238224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:45.238161    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7" podStartSLOduration=2.238143385 podStartE2EDuration="2.238143385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:45.236280458 +0000 UTC m=+2011.728097029" watchObservedRunningTime="2026-10-30 21:10:45.238143385 +0000 UTC m=+2011.729959932"
Oct 30 21:10:49.284136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.284102    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"]
Oct 30 21:10:49.289136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.289116    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:10:49.295014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.294993    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-00909f\""
Oct 30 21:10:49.295949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.295926    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-00909f-dockercfg-qxxjm\""
Oct 30 21:10:49.296283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.296264    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"]
Oct 30 21:10:49.419068 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.396769    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6\" (UID: \"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:10:49.419068 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.397058    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p2ct\" (UniqueName: \"kubernetes.io/projected/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-kube-api-access-9p2ct\") pod \"payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6\" (UID: \"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:10:49.537935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.537058    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6\" (UID: \"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:10:49.537935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.537241    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p2ct\" (UniqueName: \"kubernetes.io/projected/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-kube-api-access-9p2ct\") pod \"payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6\" (UID: \"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:10:49.547878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.538679    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6\" (UID: \"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:10:49.549597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.549517    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p2ct\" (UniqueName: \"kubernetes.io/projected/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-kube-api-access-9p2ct\") pod \"payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6\" (UID: \"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:10:49.608128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.608088    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:10:49.911404 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:10:49.911372    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0d6e1c_9bc7_4a21_9e93_aeafa944b6a1.slice/crio-15938a39bff74c3f598ae1f29090c8c2dc92e575f01f908a4e5f898b13d98c2a WatchSource:0}: Error finding container 15938a39bff74c3f598ae1f29090c8c2dc92e575f01f908a4e5f898b13d98c2a: Status 404 returned error can't find the container with id 15938a39bff74c3f598ae1f29090c8c2dc92e575f01f908a4e5f898b13d98c2a
Oct 30 21:10:49.912508 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:49.912477    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"]
Oct 30 21:10:50.247463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:50.247369    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6" event={"ID":"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1","Type":"ContainerStarted","Data":"542c215b15de66e055e780fbc4d641ac62c45950d66c7ac92871400bd0221fa7"}
Oct 30 21:10:50.247463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:50.247458    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6" event={"ID":"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1","Type":"ContainerStarted","Data":"15938a39bff74c3f598ae1f29090c8c2dc92e575f01f908a4e5f898b13d98c2a"}
Oct 30 21:10:50.247660 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:50.247479    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:10:50.279887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:50.269003    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6" podStartSLOduration=1.268986084 podStartE2EDuration="1.268986084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:50.264373687 +0000 UTC m=+2016.756190232" watchObservedRunningTime="2026-10-30 21:10:50.268986084 +0000 UTC m=+2016.760802631"
Oct 30 21:10:53.030134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:53.030098    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"]
Oct 30 21:10:53.035116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:53.035092    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140c620d-kzx8x"]
Oct 30 21:10:53.665353 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:53.665319    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67536d32-e34c-4b6e-807f-8b1b630ce07f" path="/var/lib/kubelet/pods/67536d32-e34c-4b6e-807f-8b1b630ce07f/volumes"
Oct 30 21:10:55.157481 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:10:55.157399    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:10:56.245888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:56.240124    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:10:56.254063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:56.254027    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:10:57.111689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.111655    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"]
Oct 30 21:10:57.118842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.118814    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.124244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.124164    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"openshift-service-ca.crt\""
Oct 30 21:10:57.124244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.124164    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"e2e-cmp-legacy-b4a6a0-provider-cred\""
Oct 30 21:10:57.124244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.124227    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"routing-overlay\""
Oct 30 21:10:57.125173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.125144    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99668e\""
Oct 30 21:10:57.125375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.125350    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"kube-root-ca.crt\""
Oct 30 21:10:57.125446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.125352    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"payload-processing-external-model-e2e--789fdc40-dockercfg-45z5p\""
Oct 30 21:10:57.131106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.131083    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"]
Oct 30 21:10:57.171318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.171278    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.171561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.171511    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfr8r\" (UniqueName: \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-kube-api-access-hfr8r\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.171709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.171679    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.171849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.171752    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.273764 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.273715    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.274316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.273941    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfr8r\" (UniqueName: \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-kube-api-access-hfr8r\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.274316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.274120    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.274316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.274174    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.275202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.275163    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.280885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.275796    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.293756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.293717    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfr8r\" (UniqueName: \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-kube-api-access-hfr8r\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.308500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.300696    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.437000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.436954    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:57.725060 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:10:57.725026    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda0cd71_fd0b_43b1_af47_2b8120aaa372.slice/crio-5ebaffc1d381f83ef9f63d2feb8abbd3c76c4c0d495da497b28d2c181f2b4dc4 WatchSource:0}: Error finding container 5ebaffc1d381f83ef9f63d2feb8abbd3c76c4c0d495da497b28d2c181f2b4dc4: Status 404 returned error can't find the container with id 5ebaffc1d381f83ef9f63d2feb8abbd3c76c4c0d495da497b28d2c181f2b4dc4
Oct 30 21:10:57.725180 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:57.725091    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"]
Oct 30 21:10:58.414521 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:58.414483    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx" event={"ID":"dda0cd71-fd0b-43b1-af47-2b8120aaa372","Type":"ContainerStarted","Data":"671279effd1eb8f2fd8077641c46192e0248ab33a4d283ea744e36f5f4ab7ef7"}
Oct 30 21:10:58.415046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:58.414548    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:10:58.415046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:58.414567    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx" event={"ID":"dda0cd71-fd0b-43b1-af47-2b8120aaa372","Type":"ContainerStarted","Data":"5ebaffc1d381f83ef9f63d2feb8abbd3c76c4c0d495da497b28d2c181f2b4dc4"}
Oct 30 21:10:58.435676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:10:58.435629    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx" podStartSLOduration=1.435616394 podStartE2EDuration="1.435616394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:58.43267628 +0000 UTC m=+2024.924492849" watchObservedRunningTime="2026-10-30 21:10:58.435616394 +0000 UTC m=+2024.927432938"
Oct 30 21:11:00.055128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:00.055092    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:11:01.259592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:01.259560    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:11:03.031285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:03.031251    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"]
Oct 30 21:11:03.035807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:03.035779    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba83cc37-jzv6k"]
Oct 30 21:11:03.679565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:03.679534    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca399a2-5ae1-4dd6-836d-bf84cb03549e" path="/var/lib/kubelet/pods/cca399a2-5ae1-4dd6-836d-bf84cb03549e/volumes"
Oct 30 21:11:08.656820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.656783    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"]
Oct 30 21:11:08.663292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.663257    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.673878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.671640    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"]
Oct 30 21:11:08.887225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.887083    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.887225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.887200    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmv8d\" (UniqueName: \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-kube-api-access-mmv8d\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.887639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.887314    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.887639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.887362    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.988635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.988548    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.988635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.988608    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.988880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.988824    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.988949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.988927    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmv8d\" (UniqueName: \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-kube-api-access-mmv8d\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.989438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.989411    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.989571 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.989546    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:08.991529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:08.991502    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:09.001160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:09.001130    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmv8d\" (UniqueName: \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-kube-api-access-mmv8d\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:09.067433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:09.067391    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:09.370368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:09.370342    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"]
Oct 30 21:11:09.370532 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:11:09.370495    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5ec1f6_fadf_46bd_971c_e3c727bc4957.slice/crio-013a84be2197ef78d151e9d992721798e41971f06de680fbb63a8ffac4865939 WatchSource:0}: Error finding container 013a84be2197ef78d151e9d992721798e41971f06de680fbb63a8ffac4865939: Status 404 returned error can't find the container with id 013a84be2197ef78d151e9d992721798e41971f06de680fbb63a8ffac4865939
Oct 30 21:11:09.425559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:09.425538    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:11:09.546385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:09.546349    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk" event={"ID":"8c5ec1f6-fadf-46bd-971c-e3c727bc4957","Type":"ContainerStarted","Data":"af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867"}
Oct 30 21:11:09.546559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:09.546411    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk" event={"ID":"8c5ec1f6-fadf-46bd-971c-e3c727bc4957","Type":"ContainerStarted","Data":"013a84be2197ef78d151e9d992721798e41971f06de680fbb63a8ffac4865939"}
Oct 30 21:11:09.546559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:09.546439    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:09.568399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:09.568330    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk" podStartSLOduration=1.5683156660000002 podStartE2EDuration="1.568315666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:09.564692208 +0000 UTC m=+2036.056508753" watchObservedRunningTime="2026-10-30 21:11:09.568315666 +0000 UTC m=+2036.060132211"
Oct 30 21:11:20.558876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:20.558827    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:20.611887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:20.609747    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"]
Oct 30 21:11:22.049878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.039908    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"]
Oct 30 21:11:22.049878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.047229    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"]
Oct 30 21:11:22.049878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.047320    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.059895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.054202    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.059895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.054264    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.059895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.054291    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.059895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.054348    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w77f4\" (UniqueName: \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-kube-api-access-w77f4\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.155439 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.155402    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.155635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.155449    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.155635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.155468    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.155635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.155506    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w77f4\" (UniqueName: \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-kube-api-access-w77f4\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.156145 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.156118    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.156236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.156217    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.157983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.157963    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.163655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.163632    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w77f4\" (UniqueName: \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-kube-api-access-w77f4\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.407537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.407500    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:22.699932 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:11:22.699892    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04be9db4_ee48_4260_bfd3_a1f96e22ff08.slice/crio-f0775cef5bbe82b0e25c839f50b2b40014fdae4dd9010b1a2b1cb3ddc3380d71 WatchSource:0}: Error finding container f0775cef5bbe82b0e25c839f50b2b40014fdae4dd9010b1a2b1cb3ddc3380d71: Status 404 returned error can't find the container with id f0775cef5bbe82b0e25c839f50b2b40014fdae4dd9010b1a2b1cb3ddc3380d71
Oct 30 21:11:22.702449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.702409    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"]
Oct 30 21:11:22.731221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:22.731193    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29" event={"ID":"04be9db4-ee48-4260-bfd3-a1f96e22ff08","Type":"ContainerStarted","Data":"f0775cef5bbe82b0e25c839f50b2b40014fdae4dd9010b1a2b1cb3ddc3380d71"}
Oct 30 21:11:23.740507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:23.740470    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29" event={"ID":"04be9db4-ee48-4260-bfd3-a1f96e22ff08","Type":"ContainerStarted","Data":"2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5"}
Oct 30 21:11:23.740925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:23.740536    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:23.760985 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:23.760929    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29" podStartSLOduration=2.760913638 podStartE2EDuration="2.760913638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:23.75636323 +0000 UTC m=+2050.248179774" watchObservedRunningTime="2026-10-30 21:11:23.760913638 +0000 UTC m=+2050.252730183"
Oct 30 21:11:25.157425 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:11:25.157387    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:11:25.610276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:25.610168    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g" podUID="cb31954e-6cf1-4bc4-b878-23dda9ca4b0c" containerName="payload-processing" containerID="cri-o://8768f92d665dd10d7ef01049b8cfd4143466c223c4479b557782cb4a5b24d566" gracePeriod=25
Oct 30 21:11:25.832135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:25.831919    4810 generic.go:356] "Generic (PLEG): container finished" podID="cb31954e-6cf1-4bc4-b878-23dda9ca4b0c" containerID="8768f92d665dd10d7ef01049b8cfd4143466c223c4479b557782cb4a5b24d566" exitCode=0
Oct 30 21:11:25.832135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:25.832059    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g" event={"ID":"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c","Type":"ContainerDied","Data":"8768f92d665dd10d7ef01049b8cfd4143466c223c4479b557782cb4a5b24d566"}
Oct 30 21:11:26.128971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.128933    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:11:26.284041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.283950    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-routing-overlay-volume\") pod \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") "
Oct 30 21:11:26.284524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.284046    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-provider-credentials\") pod \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") "
Oct 30 21:11:26.284524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.284175    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-plugins-config-volume\") pod \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") "
Oct 30 21:11:26.284524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.284316    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-routing-overlay-volume" pod "cb31954e-6cf1-4bc4-b878-23dda9ca4b0c" (UID: "cb31954e-6cf1-4bc4-b878-23dda9ca4b0c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:26.284524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.284467    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-kube-api-access-tb5s7\" (UniqueName: \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-kube-api-access-tb5s7\") pod \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\" (UID: \"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c\") "
Oct 30 21:11:26.284814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.284787    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-plugins-config-volume" pod "cb31954e-6cf1-4bc4-b878-23dda9ca4b0c" (UID: "cb31954e-6cf1-4bc4-b878-23dda9ca4b0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:26.285271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.285248    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:26.285398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.285278    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:26.286820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.286600    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-provider-credentials" pod "cb31954e-6cf1-4bc4-b878-23dda9ca4b0c" (UID: "cb31954e-6cf1-4bc4-b878-23dda9ca4b0c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:26.286820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.286760    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-kube-api-access-tb5s7" pod "cb31954e-6cf1-4bc4-b878-23dda9ca4b0c" (UID: "cb31954e-6cf1-4bc4-b878-23dda9ca4b0c"). InnerVolumeSpecName "kube-api-access-tb5s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:26.336044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.336010    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"]
Oct 30 21:11:26.336823 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.336803    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cb31954e-6cf1-4bc4-b878-23dda9ca4b0c" containerName="payload-processing"
Oct 30 21:11:26.336823 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.336822    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb31954e-6cf1-4bc4-b878-23dda9ca4b0c" containerName="payload-processing"
Oct 30 21:11:26.337009 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.336999    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb31954e-6cf1-4bc4-b878-23dda9ca4b0c" containerName="payload-processing"
Oct 30 21:11:26.341193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.341171    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.348205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.348166    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"routing-overlay\""
Oct 30 21:11:26.351158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.351131    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"openshift-service-ca.crt\""
Oct 30 21:11:26.351291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.351147    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"agc-cred-be61d10a5f0a\""
Oct 30 21:11:26.351291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.351170    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e1e0f5\""
Oct 30 21:11:26.351291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.351232    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"kube-root-ca.crt\""
Oct 30 21:11:26.353768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.353746    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"]
Oct 30 21:11:26.386911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.386847    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb5s7\" (UniqueName: \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-kube-api-access-tb5s7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:26.387052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.386915    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:26.522439 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.500980    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.522439 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.501083    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.522439 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.501127    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dffs\" (UniqueName: \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-kube-api-access-9dffs\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.522439 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.501312    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.610941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.607825    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.610941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.607947    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.610941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.607990    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dffs\" (UniqueName: \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-kube-api-access-9dffs\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.610941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.608171    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.610941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.609177    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.610941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.609900    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.626886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.625660    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.653977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.653947    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dffs\" (UniqueName: \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-kube-api-access-9dffs\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.659230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.659189    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:26.841456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.841421    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"
Oct 30 21:11:26.841643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.841458    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g" event={"ID":"cb31954e-6cf1-4bc4-b878-23dda9ca4b0c","Type":"ContainerDied","Data":"490bbabd8324f3dcd4c545bc682016f8344181780d05ac12342ebf9b6476b6a3"}
Oct 30 21:11:26.841643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.841519    4810 scope.go:122] "RemoveContainer" containerID="8768f92d665dd10d7ef01049b8cfd4143466c223c4479b557782cb4a5b24d566"
Oct 30 21:11:26.868949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.868835    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"]
Oct 30 21:11:26.872784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.872758    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-6644r7n8g"]
Oct 30 21:11:26.965600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:26.961640    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"]
Oct 30 21:11:26.970787 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:11:26.970764    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf83ddb_6619_4e59_a82c_dc90d9eb0f02.slice/crio-e173471e6ca383d7f31977667ab2bff99cc378f7a156793b22687459a0426fe5 WatchSource:0}: Error finding container e173471e6ca383d7f31977667ab2bff99cc378f7a156793b22687459a0426fe5: Status 404 returned error can't find the container with id e173471e6ca383d7f31977667ab2bff99cc378f7a156793b22687459a0426fe5
Oct 30 21:11:27.666996 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:27.666961    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb31954e-6cf1-4bc4-b878-23dda9ca4b0c" path="/var/lib/kubelet/pods/cb31954e-6cf1-4bc4-b878-23dda9ca4b0c/volumes"
Oct 30 21:11:27.867230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:27.867185    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz" event={"ID":"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02","Type":"ContainerStarted","Data":"244ed0a39ffb094205b5d90ba3fee8acca34649f2d9a6c2a98c921cb60dacc99"}
Oct 30 21:11:27.867446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:27.867265    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz" event={"ID":"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02","Type":"ContainerStarted","Data":"e173471e6ca383d7f31977667ab2bff99cc378f7a156793b22687459a0426fe5"}
Oct 30 21:11:27.867446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:27.867416    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:27.907880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:27.907290    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz" podStartSLOduration=1.907270385 podStartE2EDuration="1.907270385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:27.892903446 +0000 UTC m=+2054.384719993" watchObservedRunningTime="2026-10-30 21:11:27.907270385 +0000 UTC m=+2054.399086935"
Oct 30 21:11:32.067116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.067080    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"]
Oct 30 21:11:32.070937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.070912    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:32.101558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.101524    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"]
Oct 30 21:11:32.107543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.107521    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.120637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.120613    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"]
Oct 30 21:11:32.137671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.137645    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a574c2\"/\"agc-cred-b87682c5e55d\""
Oct 30 21:11:32.213692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.213648    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.213910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.213836    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.214062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.214047    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqdsv\" (UniqueName: \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-kube-api-access-gqdsv\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.214103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.214089    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.315022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.314989    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.315209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.315099    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.315284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.315226    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqdsv\" (UniqueName: \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-kube-api-access-gqdsv\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.315284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.315262    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.315949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.315922    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.316066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.316050    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.317564 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.317505    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.325077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.325053    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqdsv\" (UniqueName: \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-kube-api-access-gqdsv\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.448099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.448030    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.765469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.765434    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"]
Oct 30 21:11:32.769321 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:11:32.769296    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c940b8d_16eb_4c27_89f5_cf93b7d9958d.slice/crio-57c6b2bdb577857a0cca6238ea6ac572f69985d5ee0c7168abf5f302f6de12a2 WatchSource:0}: Error finding container 57c6b2bdb577857a0cca6238ea6ac572f69985d5ee0c7168abf5f302f6de12a2: Status 404 returned error can't find the container with id 57c6b2bdb577857a0cca6238ea6ac572f69985d5ee0c7168abf5f302f6de12a2
Oct 30 21:11:32.914207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.914161    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956" event={"ID":"6c940b8d-16eb-4c27-89f5-cf93b7d9958d","Type":"ContainerStarted","Data":"2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5"}
Oct 30 21:11:32.914385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.914226    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956" event={"ID":"6c940b8d-16eb-4c27-89f5-cf93b7d9958d","Type":"ContainerStarted","Data":"57c6b2bdb577857a0cca6238ea6ac572f69985d5ee0c7168abf5f302f6de12a2"}
Oct 30 21:11:32.914385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.914256    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:32.935633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:32.935585    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956" podStartSLOduration=0.935572256 podStartE2EDuration="935.572256ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:32.930808211 +0000 UTC m=+2059.422624768" watchObservedRunningTime="2026-10-30 21:11:32.935572256 +0000 UTC m=+2059.427388800"
Oct 30 21:11:35.969660 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:35.969624    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"]
Oct 30 21:11:35.975571 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:35.975546    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:11:35.983310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:35.983276    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f147de-dockercfg-479xs\""
Oct 30 21:11:35.983477 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:35.983360    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f147de\""
Oct 30 21:11:35.988343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:35.988259    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"]
Oct 30 21:11:36.006185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.006148    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7txmn\" (UniqueName: \"kubernetes.io/projected/5491ea4b-00ab-4989-8198-33d1f26b5a8f-kube-api-access-7txmn\") pod \"payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w\" (UID: \"5491ea4b-00ab-4989-8198-33d1f26b5a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:11:36.006336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.006208    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5491ea4b-00ab-4989-8198-33d1f26b5a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w\" (UID: \"5491ea4b-00ab-4989-8198-33d1f26b5a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:11:36.111004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.107831    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7txmn\" (UniqueName: \"kubernetes.io/projected/5491ea4b-00ab-4989-8198-33d1f26b5a8f-kube-api-access-7txmn\") pod \"payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w\" (UID: \"5491ea4b-00ab-4989-8198-33d1f26b5a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:11:36.111004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.107959    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5491ea4b-00ab-4989-8198-33d1f26b5a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w\" (UID: \"5491ea4b-00ab-4989-8198-33d1f26b5a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:11:36.111004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.109192    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5491ea4b-00ab-4989-8198-33d1f26b5a8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w\" (UID: \"5491ea4b-00ab-4989-8198-33d1f26b5a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:11:36.178921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.169109    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7txmn\" (UniqueName: \"kubernetes.io/projected/5491ea4b-00ab-4989-8198-33d1f26b5a8f-kube-api-access-7txmn\") pod \"payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w\" (UID: \"5491ea4b-00ab-4989-8198-33d1f26b5a8f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:11:36.301928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.298410    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:11:36.607436 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.607404    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"]
Oct 30 21:11:36.608165 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:11:36.608137    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5491ea4b_00ab_4989_8198_33d1f26b5a8f.slice/crio-fdd529027379f0dc78f13489e956b35363fcb0feab75c9396e23eeedaa54c210 WatchSource:0}: Error finding container fdd529027379f0dc78f13489e956b35363fcb0feab75c9396e23eeedaa54c210: Status 404 returned error can't find the container with id fdd529027379f0dc78f13489e956b35363fcb0feab75c9396e23eeedaa54c210
Oct 30 21:11:36.954081 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.954044    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w" event={"ID":"5491ea4b-00ab-4989-8198-33d1f26b5a8f","Type":"ContainerStarted","Data":"c7ba3199280d4186a957d84f953f418a73576d8481cc4ae9c1340a7507103d4e"}
Oct 30 21:11:36.954297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.954110    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w" event={"ID":"5491ea4b-00ab-4989-8198-33d1f26b5a8f","Type":"ContainerStarted","Data":"fdd529027379f0dc78f13489e956b35363fcb0feab75c9396e23eeedaa54c210"}
Oct 30 21:11:36.954297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.954169    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:11:36.975533 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:36.975483    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w" podStartSLOduration=1.975465364 podStartE2EDuration="1.975465364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:36.970760297 +0000 UTC m=+2063.462576855" watchObservedRunningTime="2026-10-30 21:11:36.975465364 +0000 UTC m=+2063.467281912"
Oct 30 21:11:37.067415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.067375    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29" podUID="04be9db4-ee48-4260-bfd3-a1f96e22ff08" containerName="payload-processing" containerID="cri-o://2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5" gracePeriod=25
Oct 30 21:11:37.359045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.359002    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:37.427773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.427728    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-provider-credentials\" (UniqueName: \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-provider-credentials\") pod \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") "
Oct 30 21:11:37.427983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.427810    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-plugins-config-volume\") pod \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") "
Oct 30 21:11:37.427983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.427965    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-kube-api-access-w77f4\" (UniqueName: \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-kube-api-access-w77f4\") pod \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") "
Oct 30 21:11:37.428198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.428175    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-routing-overlay-volume\") pod \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\" (UID: \"04be9db4-ee48-4260-bfd3-a1f96e22ff08\") "
Oct 30 21:11:37.428325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.428288    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-plugins-config-volume" pod "04be9db4-ee48-4260-bfd3-a1f96e22ff08" (UID: "04be9db4-ee48-4260-bfd3-a1f96e22ff08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:37.428537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.428514    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-routing-overlay-volume" pod "04be9db4-ee48-4260-bfd3-a1f96e22ff08" (UID: "04be9db4-ee48-4260-bfd3-a1f96e22ff08"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:37.429106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.429078    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:37.429217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.429110    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04be9db4-ee48-4260-bfd3-a1f96e22ff08-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:37.430209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.430182    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-provider-credentials" pod "04be9db4-ee48-4260-bfd3-a1f96e22ff08" (UID: "04be9db4-ee48-4260-bfd3-a1f96e22ff08"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:37.430575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.430554    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-kube-api-access-w77f4" pod "04be9db4-ee48-4260-bfd3-a1f96e22ff08" (UID: "04be9db4-ee48-4260-bfd3-a1f96e22ff08"). InnerVolumeSpecName "kube-api-access-w77f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:37.530652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.530571    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:37.530652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.530605    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w77f4\" (UniqueName: \"kubernetes.io/projected/04be9db4-ee48-4260-bfd3-a1f96e22ff08-kube-api-access-w77f4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:37.963256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.963222    4810 generic.go:356] "Generic (PLEG): container finished" podID="04be9db4-ee48-4260-bfd3-a1f96e22ff08" containerID="2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5" exitCode=0
Oct 30 21:11:37.963426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.963269    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"
Oct 30 21:11:37.963426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.963297    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29" event={"ID":"04be9db4-ee48-4260-bfd3-a1f96e22ff08","Type":"ContainerDied","Data":"2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5"}
Oct 30 21:11:37.963426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.963354    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29" event={"ID":"04be9db4-ee48-4260-bfd3-a1f96e22ff08","Type":"ContainerDied","Data":"f0775cef5bbe82b0e25c839f50b2b40014fdae4dd9010b1a2b1cb3ddc3380d71"}
Oct 30 21:11:37.963426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.963381    4810 scope.go:122] "RemoveContainer" containerID="2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5"
Oct 30 21:11:37.975958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.975935    4810 scope.go:122] "RemoveContainer" containerID="2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5"
Oct 30 21:11:37.976339 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:11:37.976254    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5\": container with ID starting with 2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5 not found: ID does not exist" containerID="2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5"
Oct 30 21:11:37.976339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.976279    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5"} err="failed to get container status \"2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5\": rpc error: code = NotFound desc = could not find container \"2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5\": container with ID starting with 2f7058598a1f757b080da6fe4a9ab3b0222cbb671439fe185393d4a5b3bb0fe5 not found: ID does not exist"
Oct 30 21:11:37.985774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.985743    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"]
Oct 30 21:11:37.989744 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:37.989720    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49ch4m29"]
Oct 30 21:11:38.879215 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:38.879175    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:11:39.664828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:39.664795    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04be9db4-ee48-4260-bfd3-a1f96e22ff08" path="/var/lib/kubelet/pods/04be9db4-ee48-4260-bfd3-a1f96e22ff08/volumes"
Oct 30 21:11:43.927188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:43.927144    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:11:43.984192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:43.984147    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"]
Oct 30 21:11:44.762134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:44.762102    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:11:47.967162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:47.967134    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:11:48.985106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:48.985058    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk" podUID="8c5ec1f6-fadf-46bd-971c-e3c727bc4957" containerName="payload-processing" containerID="cri-o://af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867" gracePeriod=25
Oct 30 21:11:49.312203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.312172    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:49.346117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.346077    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-plugins-config-volume\") pod \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") "
Oct 30 21:11:49.346435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.346246    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-kube-api-access-mmv8d\" (UniqueName: \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-kube-api-access-mmv8d\") pod \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") "
Oct 30 21:11:49.346435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.346301    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-routing-overlay-volume\") pod \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") "
Oct 30 21:11:49.346435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.346339    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-provider-credentials\") pod \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\" (UID: \"8c5ec1f6-fadf-46bd-971c-e3c727bc4957\") "
Oct 30 21:11:49.347024 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.346993    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-routing-overlay-volume" pod "8c5ec1f6-fadf-46bd-971c-e3c727bc4957" (UID: "8c5ec1f6-fadf-46bd-971c-e3c727bc4957"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:49.347523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.347474    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-plugins-config-volume" pod "8c5ec1f6-fadf-46bd-971c-e3c727bc4957" (UID: "8c5ec1f6-fadf-46bd-971c-e3c727bc4957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:49.349687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.349653    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-provider-credentials" pod "8c5ec1f6-fadf-46bd-971c-e3c727bc4957" (UID: "8c5ec1f6-fadf-46bd-971c-e3c727bc4957"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:49.349891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.349835    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-kube-api-access-mmv8d" pod "8c5ec1f6-fadf-46bd-971c-e3c727bc4957" (UID: "8c5ec1f6-fadf-46bd-971c-e3c727bc4957"). InnerVolumeSpecName "kube-api-access-mmv8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:49.450494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.450386    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.450494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.450432    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmv8d\" (UniqueName: \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-kube-api-access-mmv8d\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.450494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.450449    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.450494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:49.450463    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c5ec1f6-fadf-46bd-971c-e3c727bc4957-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:50.086731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:50.086644    4810 generic.go:356] "Generic (PLEG): container finished" podID="8c5ec1f6-fadf-46bd-971c-e3c727bc4957" containerID="af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867" exitCode=0
Oct 30 21:11:50.087212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:50.086723    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"
Oct 30 21:11:50.087212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:50.086720    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk" event={"ID":"8c5ec1f6-fadf-46bd-971c-e3c727bc4957","Type":"ContainerDied","Data":"af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867"}
Oct 30 21:11:50.087212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:50.086884    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk" event={"ID":"8c5ec1f6-fadf-46bd-971c-e3c727bc4957","Type":"ContainerDied","Data":"013a84be2197ef78d151e9d992721798e41971f06de680fbb63a8ffac4865939"}
Oct 30 21:11:50.087212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:50.086907    4810 scope.go:122] "RemoveContainer" containerID="af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867"
Oct 30 21:11:50.100738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:50.100717    4810 scope.go:122] "RemoveContainer" containerID="af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867"
Oct 30 21:11:50.101057 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:11:50.101029    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867\": container with ID starting with af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867 not found: ID does not exist" containerID="af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867"
Oct 30 21:11:50.101134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:50.101068    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867"} err="failed to get container status \"af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867\": rpc error: code = NotFound desc = could not find container \"af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867\": container with ID starting with af76e2295ca4ed97f7d6a62a3606de2257d5f6a62f043b0f5536aaec23d67867 not found: ID does not exist"
Oct 30 21:11:50.109107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:50.109082    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"]
Oct 30 21:11:50.113808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:50.113786    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f868zlgfk"]
Oct 30 21:11:51.665416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:11:51.665376    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c5ec1f6-fadf-46bd-971c-e3c727bc4957" path="/var/lib/kubelet/pods/8c5ec1f6-fadf-46bd-971c-e3c727bc4957/volumes"
Oct 30 21:11:55.157356 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:11:55.157323    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:12:05.814635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.814601    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"]
Oct 30 21:12:05.815606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.815581    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8c5ec1f6-fadf-46bd-971c-e3c727bc4957" containerName="payload-processing"
Oct 30 21:12:05.815720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.815606    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c5ec1f6-fadf-46bd-971c-e3c727bc4957" containerName="payload-processing"
Oct 30 21:12:05.815720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.815664    4810 cpu_manager.go:414] "containerMap: removing container" podUID="04be9db4-ee48-4260-bfd3-a1f96e22ff08" containerName="payload-processing"
Oct 30 21:12:05.815720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.815673    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04be9db4-ee48-4260-bfd3-a1f96e22ff08" containerName="payload-processing"
Oct 30 21:12:05.815941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.815926    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c5ec1f6-fadf-46bd-971c-e3c727bc4957" containerName="payload-processing"
Oct 30 21:12:05.816013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.815956    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="04be9db4-ee48-4260-bfd3-a1f96e22ff08" containerName="payload-processing"
Oct 30 21:12:05.819585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.819566    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:05.827184 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.827159    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"agc-cred-2e9d66610926\""
Oct 30 21:12:05.830567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.827492    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"payload-processing-external-model-e2e--8ac6fff3-dockercfg-v7d4n\""
Oct 30 21:12:05.839317 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:05.833115    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"]
Oct 30 21:12:06.004626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.004579    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.004870 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.004740    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r659j\" (UniqueName: \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-kube-api-access-r659j\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.005000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.004977    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.005124 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.005105    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.106420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.106328    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.106420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.106396    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.106661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.106488    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r659j\" (UniqueName: \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-kube-api-access-r659j\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.106661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.106621    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.107302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.107280    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.107406 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.107377    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.109303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.109275    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.115449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.115423    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r659j\" (UniqueName: \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-kube-api-access-r659j\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-574css4lw\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.153468 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.153440    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:06.442388 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:06.442361    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"]
Oct 30 21:12:06.443346 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:12:06.443314    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb43ea3e_4370_4b40_b946_32938c66f554.slice/crio-6e30629a0ce520681f7b74566a8c5f3f398dac402b7e9a3cfe792279cfbfcd8c WatchSource:0}: Error finding container 6e30629a0ce520681f7b74566a8c5f3f398dac402b7e9a3cfe792279cfbfcd8c: Status 404 returned error can't find the container with id 6e30629a0ce520681f7b74566a8c5f3f398dac402b7e9a3cfe792279cfbfcd8c
Oct 30 21:12:07.247850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:07.247812    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw" event={"ID":"fb43ea3e-4370-4b40-b946-32938c66f554","Type":"ContainerStarted","Data":"24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a"}
Oct 30 21:12:07.248366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:07.247883    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw" event={"ID":"fb43ea3e-4370-4b40-b946-32938c66f554","Type":"ContainerStarted","Data":"6e30629a0ce520681f7b74566a8c5f3f398dac402b7e9a3cfe792279cfbfcd8c"}
Oct 30 21:12:07.248366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:07.247955    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:07.269160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:07.269111    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw" podStartSLOduration=2.269099468 podStartE2EDuration="2.269099468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:07.264665945 +0000 UTC m=+2093.756482492" watchObservedRunningTime="2026-10-30 21:12:07.269099468 +0000 UTC m=+2093.760916013"
Oct 30 21:12:13.027519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:13.027480    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"]
Oct 30 21:12:13.031705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:13.031677    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09750a7-k65th"]
Oct 30 21:12:13.664253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:13.664225    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c15b3a-53dd-4554-8141-b8a1ec58727e" path="/var/lib/kubelet/pods/57c15b3a-53dd-4554-8141-b8a1ec58727e/volumes"
Oct 30 21:12:14.031162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:14.031070    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"]
Oct 30 21:12:14.035438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:14.035410    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5775d53-dnfcc"]
Oct 30 21:12:14.432602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:14.432576    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:12:14.469666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:14.469643    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:12:15.684998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:15.684958    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="354df1b9-f265-40bf-bb6a-0e3bffa8dbb6" path="/var/lib/kubelet/pods/354df1b9-f265-40bf-bb6a-0e3bffa8dbb6/volumes"
Oct 30 21:12:16.164031 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.163990    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"]
Oct 30 21:12:16.170300 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.170272    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.178907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.178838    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"]
Oct 30 21:12:16.265437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.265405    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:16.265588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.265463    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:12:16.277577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.277553    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e3f8913-kmxrc\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.277732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.277590    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e3f8913-kmxrc\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.277732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.277634    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87zx\" (UniqueName: \"kubernetes.io/projected/65d19158-93cf-4b69-bf7c-46f7d3d849dd-kube-api-access-b87zx\") pod \"maas-api-revoke-sub-6e3f8913-kmxrc\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.379263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.379227    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e3f8913-kmxrc\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.379450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.379275    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e3f8913-kmxrc\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.379450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.379322    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87zx\" (UniqueName: \"kubernetes.io/projected/65d19158-93cf-4b69-bf7c-46f7d3d849dd-kube-api-access-b87zx\") pod \"maas-api-revoke-sub-6e3f8913-kmxrc\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.379967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.379938    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e3f8913-kmxrc\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.381709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.381688    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e3f8913-kmxrc\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.388006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.387978    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87zx\" (UniqueName: \"kubernetes.io/projected/65d19158-93cf-4b69-bf7c-46f7d3d849dd-kube-api-access-b87zx\") pod \"maas-api-revoke-sub-6e3f8913-kmxrc\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.491910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.491795    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:16.493339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.493315    4810 scope.go:122] "RemoveContainer" containerID="c725f9668c3060066f4ee13d65a5b85cfecbbd4fd67bad3beb88fd202fd47179"
Oct 30 21:12:16.510407 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.510329    4810 scope.go:122] "RemoveContainer" containerID="30a4dda8f7198b83c7af25b23bcdd5955dfe68638a894b91db221304e4e8661d"
Oct 30 21:12:16.807107 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:12:16.807054    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d19158_93cf_4b69_bf7c_46f7d3d849dd.slice/crio-817aac7ade2d70a1d16da129e8672845ba621f80508454e275556e668ed61998 WatchSource:0}: Error finding container 817aac7ade2d70a1d16da129e8672845ba621f80508454e275556e668ed61998: Status 404 returned error can't find the container with id 817aac7ade2d70a1d16da129e8672845ba621f80508454e275556e668ed61998
Oct 30 21:12:16.807107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:16.807088    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"]
Oct 30 21:12:17.349282 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:17.349201    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc" event={"ID":"65d19158-93cf-4b69-bf7c-46f7d3d849dd","Type":"ContainerStarted","Data":"817aac7ade2d70a1d16da129e8672845ba621f80508454e275556e668ed61998"}
Oct 30 21:12:18.259226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:18.259148    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:12:18.327041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:18.327005    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"]
Oct 30 21:12:18.361260 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:18.361220    4810 generic.go:356] "Generic (PLEG): container finished" podID="65d19158-93cf-4b69-bf7c-46f7d3d849dd" containerID="9974e7b993d1f8bd43e83edb3e3a4feff0a8ede56f41e29ea5e43ee56737331b" exitCode=0
Oct 30 21:12:18.361579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:18.361408    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc" event={"ID":"65d19158-93cf-4b69-bf7c-46f7d3d849dd","Type":"ContainerDied","Data":"9974e7b993d1f8bd43e83edb3e3a4feff0a8ede56f41e29ea5e43ee56737331b"}
Oct 30 21:12:19.536132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:19.536107    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:19.619757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:19.619671    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-openshift-ca\") pod \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") "
Oct 30 21:12:19.619935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:19.619789    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d19158-93cf-4b69-bf7c-46f7d3d849dd-kube-api-access-b87zx\" (UniqueName: \"kubernetes.io/projected/65d19158-93cf-4b69-bf7c-46f7d3d849dd-kube-api-access-b87zx\") pod \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") "
Oct 30 21:12:19.619935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:19.619889    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-xks-ca\") pod \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\" (UID: \"65d19158-93cf-4b69-bf7c-46f7d3d849dd\") "
Oct 30 21:12:19.620077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:19.620055    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-openshift-ca" pod "65d19158-93cf-4b69-bf7c-46f7d3d849dd" (UID: "65d19158-93cf-4b69-bf7c-46f7d3d849dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:19.620495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:19.620462    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:19.622085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:19.622059    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d19158-93cf-4b69-bf7c-46f7d3d849dd-kube-api-access-b87zx" pod "65d19158-93cf-4b69-bf7c-46f7d3d849dd" (UID: "65d19158-93cf-4b69-bf7c-46f7d3d849dd"). InnerVolumeSpecName "kube-api-access-b87zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:19.622202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:19.622085    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-xks-ca" pod "65d19158-93cf-4b69-bf7c-46f7d3d849dd" (UID: "65d19158-93cf-4b69-bf7c-46f7d3d849dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:19.722182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:19.722143    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b87zx\" (UniqueName: \"kubernetes.io/projected/65d19158-93cf-4b69-bf7c-46f7d3d849dd-kube-api-access-b87zx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:19.722182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:19.722175    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d19158-93cf-4b69-bf7c-46f7d3d849dd-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:20.379484 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:20.379447    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc" event={"ID":"65d19158-93cf-4b69-bf7c-46f7d3d849dd","Type":"ContainerDied","Data":"817aac7ade2d70a1d16da129e8672845ba621f80508454e275556e668ed61998"}
Oct 30 21:12:20.379643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:20.379501    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="817aac7ade2d70a1d16da129e8672845ba621f80508454e275556e668ed61998"
Oct 30 21:12:20.379643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:20.379503    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"
Oct 30 21:12:23.027735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.027689    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"]
Oct 30 21:12:23.032199 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.032168    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4810c1-r2sgm"]
Oct 30 21:12:23.328310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.328219    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw" podUID="1ebfea03-bc7c-4de7-b677-220520b6ab6e" containerName="payload-processing" containerID="cri-o://4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6" gracePeriod=25
Oct 30 21:12:23.661895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.661841    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:12:23.664845 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.664823    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff847ad-b0b8-4a60-b23e-6b0c09d273e9" path="/var/lib/kubelet/pods/6ff847ad-b0b8-4a60-b23e-6b0c09d273e9/volumes"
Oct 30 21:12:23.779622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.779591    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-routing-overlay-volume\") pod \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") "
Oct 30 21:12:23.779830 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.779749    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-kube-api-access-6z8qp\" (UniqueName: \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-kube-api-access-6z8qp\") pod \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") "
Oct 30 21:12:23.779949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.779932    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-plugins-config-volume\") pod \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") "
Oct 30 21:12:23.780019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.779998    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-routing-overlay-volume" pod "1ebfea03-bc7c-4de7-b677-220520b6ab6e" (UID: "1ebfea03-bc7c-4de7-b677-220520b6ab6e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:23.780089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.780073    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-provider-credentials\") pod \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\" (UID: \"1ebfea03-bc7c-4de7-b677-220520b6ab6e\") "
Oct 30 21:12:23.780394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.780364    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-plugins-config-volume" pod "1ebfea03-bc7c-4de7-b677-220520b6ab6e" (UID: "1ebfea03-bc7c-4de7-b677-220520b6ab6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:23.780666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.780643    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:23.780745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.780673    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ebfea03-bc7c-4de7-b677-220520b6ab6e-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:23.782274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.782244    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-kube-api-access-6z8qp" pod "1ebfea03-bc7c-4de7-b677-220520b6ab6e" (UID: "1ebfea03-bc7c-4de7-b677-220520b6ab6e"). InnerVolumeSpecName "kube-api-access-6z8qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:23.782396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.782310    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-provider-credentials" pod "1ebfea03-bc7c-4de7-b677-220520b6ab6e" (UID: "1ebfea03-bc7c-4de7-b677-220520b6ab6e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:23.882041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.882009    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:23.882041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:23.882036    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8qp\" (UniqueName: \"kubernetes.io/projected/1ebfea03-bc7c-4de7-b677-220520b6ab6e-kube-api-access-6z8qp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:24.025873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.025821    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"]
Oct 30 21:12:24.032636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.032602    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"]
Oct 30 21:12:24.036496 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.036470    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003e64de-qvx7l"]
Oct 30 21:12:24.039926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.039904    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c55bbb2e-chk9z"]
Oct 30 21:12:24.413958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.413926    4810 generic.go:356] "Generic (PLEG): container finished" podID="1ebfea03-bc7c-4de7-b677-220520b6ab6e" containerID="4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6" exitCode=0
Oct 30 21:12:24.414164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.413985    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"
Oct 30 21:12:24.414164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.414016    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw" event={"ID":"1ebfea03-bc7c-4de7-b677-220520b6ab6e","Type":"ContainerDied","Data":"4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6"}
Oct 30 21:12:24.414164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.414071    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw" event={"ID":"1ebfea03-bc7c-4de7-b677-220520b6ab6e","Type":"ContainerDied","Data":"f0e757bebb2ae726e2b3e19e521c4235587efe8bf89f0cc73d785b220cee23c2"}
Oct 30 21:12:24.414164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.414094    4810 scope.go:122] "RemoveContainer" containerID="4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6"
Oct 30 21:12:24.425951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.425931    4810 scope.go:122] "RemoveContainer" containerID="4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6"
Oct 30 21:12:24.426227 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:24.426210    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6\": container with ID starting with 4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6 not found: ID does not exist" containerID="4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6"
Oct 30 21:12:24.426283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.426235    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6"} err="failed to get container status \"4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6\": rpc error: code = NotFound desc = could not find container \"4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6\": container with ID starting with 4f478d5c3acaafa9f4ecc9033c0db2a04983bfeaf69da81aecbdc9ace97873e6 not found: ID does not exist"
Oct 30 21:12:24.442522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.442489    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"]
Oct 30 21:12:24.446323 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:24.446303    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bqbktw"]
Oct 30 21:12:25.156576 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:25.156544    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:12:25.663875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:25.663834    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebfea03-bc7c-4de7-b677-220520b6ab6e" path="/var/lib/kubelet/pods/1ebfea03-bc7c-4de7-b677-220520b6ab6e/volumes"
Oct 30 21:12:25.664326 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:25.664311    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5871ce-de61-4a4f-bd21-0393ef7f60bd" path="/var/lib/kubelet/pods/fb5871ce-de61-4a4f-bd21-0393ef7f60bd/volumes"
Oct 30 21:12:25.664734 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:25.664721    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8e3c84-34a0-4a16-ac5f-2fa171c35f90" path="/var/lib/kubelet/pods/fc8e3c84-34a0-4a16-ac5f-2fa171c35f90/volumes"
Oct 30 21:12:27.189383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:27.189347    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"]
Oct 30 21:12:30.153967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.153930    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"]
Oct 30 21:12:30.154764 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.154747    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1ebfea03-bc7c-4de7-b677-220520b6ab6e" containerName="payload-processing"
Oct 30 21:12:30.154819 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.154766    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebfea03-bc7c-4de7-b677-220520b6ab6e" containerName="payload-processing"
Oct 30 21:12:30.154819 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.154787    4810 cpu_manager.go:414] "containerMap: removing container" podUID="65d19158-93cf-4b69-bf7c-46f7d3d849dd" containerName="revoke-keys"
Oct 30 21:12:30.154819 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.154793    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d19158-93cf-4b69-bf7c-46f7d3d849dd" containerName="revoke-keys"
Oct 30 21:12:30.154957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.154914    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d19158-93cf-4b69-bf7c-46f7d3d849dd" containerName="revoke-keys"
Oct 30 21:12:30.154957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.154933    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebfea03-bc7c-4de7-b677-220520b6ab6e" containerName="payload-processing"
Oct 30 21:12:30.160303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.160284    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.167512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.167484    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"]
Oct 30 21:12:30.201911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.201886    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"agc-cred-fcb5e5b64832\""
Oct 30 21:12:30.202126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.202016    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"payload-processing-external-model-e2e--789fdc40-dockercfg-lbwpd\""
Oct 30 21:12:30.220064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.220035    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.220430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.220388    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.220644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.220624    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhcxl\" (UniqueName: \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-kube-api-access-bhcxl\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.220877 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.220839    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.324884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.324019    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.324884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.324171    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.324884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.324249    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhcxl\" (UniqueName: \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-kube-api-access-bhcxl\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.324884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.324361    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.328879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.328018    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.328879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.328618    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.332918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.331068    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.346878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.339013    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhcxl\" (UniqueName: \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-kube-api-access-bhcxl\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.478251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.478160    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:30.783412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:30.783377    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"]
Oct 30 21:12:31.626898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:31.626837    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" event={"ID":"d1f3dfcf-5348-4db4-a0cb-b46c6587d893","Type":"ContainerStarted","Data":"4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861"}
Oct 30 21:12:31.627389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:31.626927    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:31.627389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:31.626949    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" event={"ID":"d1f3dfcf-5348-4db4-a0cb-b46c6587d893","Type":"ContainerStarted","Data":"1c8ec9aead3f202b0da2829cdd0c805a3c622d22caa6bd396778a452cfa952c5"}
Oct 30 21:12:31.646615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:31.646567    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" podStartSLOduration=1.6465535789999999 podStartE2EDuration="1.646553579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:31.644340797 +0000 UTC m=+2118.136157339" watchObservedRunningTime="2026-10-30 21:12:31.646553579 +0000 UTC m=+2118.138370124"
Oct 30 21:12:32.191914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.190419    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx" podUID="dda0cd71-fd0b-43b1-af47-2b8120aaa372" containerName="payload-processing" containerID="cri-o://671279effd1eb8f2fd8077641c46192e0248ab33a4d283ea744e36f5f4ab7ef7" gracePeriod=25
Oct 30 21:12:32.582997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.573003    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"]
Oct 30 21:12:32.582997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.580183    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.587171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.586590    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"e2e-cfl-270aa7-provider-cred\""
Oct 30 21:12:32.603983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.590465    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"]
Oct 30 21:12:32.642801 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.642767    4810 generic.go:356] "Generic (PLEG): container finished" podID="dda0cd71-fd0b-43b1-af47-2b8120aaa372" containerID="671279effd1eb8f2fd8077641c46192e0248ab33a4d283ea744e36f5f4ab7ef7" exitCode=0
Oct 30 21:12:32.644379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.644353    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx" event={"ID":"dda0cd71-fd0b-43b1-af47-2b8120aaa372","Type":"ContainerDied","Data":"671279effd1eb8f2fd8077641c46192e0248ab33a4d283ea744e36f5f4ab7ef7"}
Oct 30 21:12:32.668481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.668456    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.668598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.668561    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.668664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.668596    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bst79\" (UniqueName: \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-kube-api-access-bst79\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.668708 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:32.668677    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-fcb5e5b64832: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:32.668708 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:32.668699    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:32.668808 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:32.668765    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials podName:d1f3dfcf-5348-4db4-a0cb-b46c6587d893 nodeName:}" failed. No retries permitted until 2026-10-30 21:12:33.168744944 +0000 UTC m=+2119.660561484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" (UID: "d1f3dfcf-5348-4db4-a0cb-b46c6587d893") : secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:32.668970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.668951    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.694118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.694092    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:12:32.770310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.770239    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-kube-api-access-hfr8r\" (UniqueName: \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-kube-api-access-hfr8r\") pod \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") "
Oct 30 21:12:32.770458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.770324    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-routing-overlay-volume\") pod \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") "
Oct 30 21:12:32.770458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.770370    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-provider-credentials\") pod \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") "
Oct 30 21:12:32.770458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.770438    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-plugins-config-volume\") pod \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\" (UID: \"dda0cd71-fd0b-43b1-af47-2b8120aaa372\") "
Oct 30 21:12:32.770678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.770654    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-routing-overlay-volume" pod "dda0cd71-fd0b-43b1-af47-2b8120aaa372" (UID: "dda0cd71-fd0b-43b1-af47-2b8120aaa372"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:32.770794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.770774    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.770884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.770838    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-plugins-config-volume" pod "dda0cd71-fd0b-43b1-af47-2b8120aaa372" (UID: "dda0cd71-fd0b-43b1-af47-2b8120aaa372"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:32.771020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.770996    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.771096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.771059    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst79\" (UniqueName: \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-kube-api-access-bst79\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.771218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.771198    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.771694 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.771647    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.772940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.772913    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-provider-credentials" pod "dda0cd71-fd0b-43b1-af47-2b8120aaa372" (UID: "dda0cd71-fd0b-43b1-af47-2b8120aaa372"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:32.773060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.773040    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-kube-api-access-hfr8r" pod "dda0cd71-fd0b-43b1-af47-2b8120aaa372" (UID: "dda0cd71-fd0b-43b1-af47-2b8120aaa372"). InnerVolumeSpecName "kube-api-access-hfr8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:32.773343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.773324    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:32.773424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.773349    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda0cd71-fd0b-43b1-af47-2b8120aaa372-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:32.773512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.773494    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.773844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.773819    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.780301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.780276    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bst79\" (UniqueName: \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-kube-api-access-bst79\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:32.874872 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.874825    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfr8r\" (UniqueName: \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-kube-api-access-hfr8r\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:32.875053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.874887    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dda0cd71-fd0b-43b1-af47-2b8120aaa372-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:32.911040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:32.911008    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:33.124048 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.123957    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:12:33.179906 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:33.179871    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-fcb5e5b64832: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:33.179906 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:33.179900    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:33.180178 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:33.179954    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials podName:d1f3dfcf-5348-4db4-a0cb-b46c6587d893 nodeName:}" failed. No retries permitted until 2026-10-30 21:12:34.179940901 +0000 UTC m=+2120.671757425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" (UID: "d1f3dfcf-5348-4db4-a0cb-b46c6587d893") : secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:33.204968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.204932    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"]
Oct 30 21:12:33.217009 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:12:33.216979    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5afc507_5879_48be_b497_34e36983685f.slice/crio-9d62c2cb51ee9997c15154b5226288e25f6d9cb35d121d6d6702b33977cbb471 WatchSource:0}: Error finding container 9d62c2cb51ee9997c15154b5226288e25f6d9cb35d121d6d6702b33977cbb471: Status 404 returned error can't find the container with id 9d62c2cb51ee9997c15154b5226288e25f6d9cb35d121d6d6702b33977cbb471
Oct 30 21:12:33.716882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.708134    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx" event={"ID":"dda0cd71-fd0b-43b1-af47-2b8120aaa372","Type":"ContainerDied","Data":"5ebaffc1d381f83ef9f63d2feb8abbd3c76c4c0d495da497b28d2c181f2b4dc4"}
Oct 30 21:12:33.716882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.708205    4810 scope.go:122] "RemoveContainer" containerID="671279effd1eb8f2fd8077641c46192e0248ab33a4d283ea744e36f5f4ab7ef7"
Oct 30 21:12:33.716882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.708365    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"
Oct 30 21:12:33.745113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.743815    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r" event={"ID":"c5afc507-5879-48be-b497-34e36983685f","Type":"ContainerStarted","Data":"51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1"}
Oct 30 21:12:33.745113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.743899    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r" event={"ID":"c5afc507-5879-48be-b497-34e36983685f","Type":"ContainerStarted","Data":"9d62c2cb51ee9997c15154b5226288e25f6d9cb35d121d6d6702b33977cbb471"}
Oct 30 21:12:33.745113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.745052    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:33.766041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.765982    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r" podStartSLOduration=1.7659616 podStartE2EDuration="1.7659616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:33.763275731 +0000 UTC m=+2120.255092276" watchObservedRunningTime="2026-10-30 21:12:33.7659616 +0000 UTC m=+2120.257778146"
Oct 30 21:12:33.782936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.782838    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"]
Oct 30 21:12:33.787070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:33.787041    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76ddbgmdx"]
Oct 30 21:12:34.198873 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:34.198825    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-fcb5e5b64832: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:34.199052 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:34.198852    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:34.199052 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:34.198935    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials podName:d1f3dfcf-5348-4db4-a0cb-b46c6587d893 nodeName:}" failed. No retries permitted until 2026-10-30 21:12:36.19892151 +0000 UTC m=+2122.690738034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" (UID: "d1f3dfcf-5348-4db4-a0cb-b46c6587d893") : secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:34.882787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.882747    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"]
Oct 30 21:12:34.900665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.893720    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dda0cd71-fd0b-43b1-af47-2b8120aaa372" containerName="payload-processing"
Oct 30 21:12:34.900665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.893751    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda0cd71-fd0b-43b1-af47-2b8120aaa372" containerName="payload-processing"
Oct 30 21:12:34.900665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.894041    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda0cd71-fd0b-43b1-af47-2b8120aaa372" containerName="payload-processing"
Oct 30 21:12:34.922779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.922742    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:34.937934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.928537    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-00909f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-00909f\""
Oct 30 21:12:34.937934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.928654    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-00909f\"/\"payload-processing-external-model-e2e--3c4e5353-dockercfg-gjtjk\""
Oct 30 21:12:34.937934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.928696    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-00909f\"/\"e2e-cmp-legacy-c51547-provider-cred\""
Oct 30 21:12:34.937934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.928758    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-00909f\"/\"routing-overlay\""
Oct 30 21:12:34.937934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.928839    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-00909f\"/\"kube-root-ca.crt\""
Oct 30 21:12:34.937934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.928988    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-00909f\"/\"openshift-service-ca.crt\""
Oct 30 21:12:34.957890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:34.943695    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"]
Oct 30 21:12:35.028510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.028468    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.028700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.028682    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.028749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.028721    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhtn\" (UniqueName: \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-kube-api-access-dwhtn\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.028822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.028810    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.132925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.130130    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.132925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.130350    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.132925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.130397    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhtn\" (UniqueName: \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-kube-api-access-dwhtn\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.132925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.130481    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.132925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.132030    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.132925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.132609    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.133896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.133829    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.151876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.151833    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhtn\" (UniqueName: \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-kube-api-access-dwhtn\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.252273 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.252230    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.566203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.566173    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"]
Oct 30 21:12:35.566942 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:12:35.566914    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c0e156_6da3_4e3d_afe1_bd28d029505c.slice/crio-c7f9d57f2835a700cbd402ad6a91d977e58ed29c9a6fdd0aed7454f08cbffb50 WatchSource:0}: Error finding container c7f9d57f2835a700cbd402ad6a91d977e58ed29c9a6fdd0aed7454f08cbffb50: Status 404 returned error can't find the container with id c7f9d57f2835a700cbd402ad6a91d977e58ed29c9a6fdd0aed7454f08cbffb50
Oct 30 21:12:35.665218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.665190    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda0cd71-fd0b-43b1-af47-2b8120aaa372" path="/var/lib/kubelet/pods/dda0cd71-fd0b-43b1-af47-2b8120aaa372/volumes"
Oct 30 21:12:35.768365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.768297    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv" event={"ID":"68c0e156-6da3-4e3d-afe1-bd28d029505c","Type":"ContainerStarted","Data":"fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f"}
Oct 30 21:12:35.768801 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.768773    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:35.768965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.768816    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv" event={"ID":"68c0e156-6da3-4e3d-afe1-bd28d029505c","Type":"ContainerStarted","Data":"c7f9d57f2835a700cbd402ad6a91d977e58ed29c9a6fdd0aed7454f08cbffb50"}
Oct 30 21:12:35.790925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:35.790847    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv" podStartSLOduration=1.790826357 podStartE2EDuration="1.790826357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:35.788749398 +0000 UTC m=+2122.280565942" watchObservedRunningTime="2026-10-30 21:12:35.790826357 +0000 UTC m=+2122.282642903"
Oct 30 21:12:36.258552 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:36.258517    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-fcb5e5b64832: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:36.258552 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:36.258544    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:36.259027 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:36.258620    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials podName:d1f3dfcf-5348-4db4-a0cb-b46c6587d893 nodeName:}" failed. No retries permitted until 2026-10-30 21:12:40.258606802 +0000 UTC m=+2126.750423326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" (UID: "d1f3dfcf-5348-4db4-a0cb-b46c6587d893") : secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:37.839005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.838972    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"]
Oct 30 21:12:37.843933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.843909    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:37.851359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.851262    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"agc-cred-437d372ad5d5\""
Oct 30 21:12:37.851359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.851296    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"openshift-service-ca.crt\""
Oct 30 21:12:37.851359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.851310    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"kube-root-ca.crt\""
Oct 30 21:12:37.851359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.851309    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6741b1\""
Oct 30 21:12:37.851359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.851310    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"routing-overlay\""
Oct 30 21:12:37.851627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.851367    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"payload-processing-external-model-e2e--a89b3f58-dockercfg-m299n\""
Oct 30 21:12:37.853311 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.853285    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"]
Oct 30 21:12:37.911872 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.911825    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:37.912046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.912029    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:37.912098 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.912067    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:37.912256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:37.912234    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhk2\" (UniqueName: \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-kube-api-access-5hhk2\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:38.025878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.025806    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:38.026068 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.025901    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:38.026161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.026132    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhk2\" (UniqueName: \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-kube-api-access-5hhk2\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:38.032542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.030799    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:38.032542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.032005    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:38.032916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.032821    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:38.046970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.044081    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"]
Oct 30 21:12:38.061829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.054351    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.061829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.056818    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhk2\" (UniqueName: \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-kube-api-access-5hhk2\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:38.061829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.060531    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:38.061829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.060742    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"agc-cred-deb2c75b9d02\""
Oct 30 21:12:38.078267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.069834    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"]
Oct 30 21:12:38.135512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.132631    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.135512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.132711    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.135512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.132786    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd869\" (UniqueName: \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-kube-api-access-wd869\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.135512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.132849    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.160997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.160140    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:38.285086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.281617    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd869\" (UniqueName: \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-kube-api-access-wd869\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.285086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.281710    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.285086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.281946    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.285086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.282013    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.285086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.282879    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.305880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.286979    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.305880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.287845    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.321588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.321517    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd869\" (UniqueName: \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-kube-api-access-wd869\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.504454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.500266    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:38.615880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:38.612270    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"]
Oct 30 21:12:38.676319 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:12:38.659192    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d459cac_0e34_4d12_91e9_ff0bfae8d20e.slice/crio-d94672ccd77ffd0b481982b4692135bac0424d835b43f2cbd8d9276a4d42404e WatchSource:0}: Error finding container d94672ccd77ffd0b481982b4692135bac0424d835b43f2cbd8d9276a4d42404e: Status 404 returned error can't find the container with id d94672ccd77ffd0b481982b4692135bac0424d835b43f2cbd8d9276a4d42404e
Oct 30 21:12:39.064354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.064257    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" event={"ID":"0d459cac-0e34-4d12-91e9-ff0bfae8d20e","Type":"ContainerStarted","Data":"d94672ccd77ffd0b481982b4692135bac0424d835b43f2cbd8d9276a4d42404e"}
Oct 30 21:12:39.302401 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:12:39.301971    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0805f44_d79c_49c0_a247_182964ec985d.slice/crio-29f17e0649d6cdea21f9e673507284ff9b17e827a26dbb728ad71dd8b0d4f73c WatchSource:0}: Error finding container 29f17e0649d6cdea21f9e673507284ff9b17e827a26dbb728ad71dd8b0d4f73c: Status 404 returned error can't find the container with id 29f17e0649d6cdea21f9e673507284ff9b17e827a26dbb728ad71dd8b0d4f73c
Oct 30 21:12:39.310000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.309242    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"]
Oct 30 21:12:39.647788 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.647747    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"]
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.668237    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.671839    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.672037    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.672111    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.672243    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnk2h\" (UniqueName: \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-kube-api-access-qnk2h\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.672789    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"agc-cred-5e55dc8ae840\""
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.672908    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"payload-processing-external-model-e2e--3ca9ce38-dockercfg-sn5pt\""
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.673532    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"routing-overlay\""
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.673595    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e8063c\""
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.673707    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"openshift-service-ca.crt\""
Oct 30 21:12:39.675908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.673793    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"kube-root-ca.crt\""
Oct 30 21:12:39.722389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.722359    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"]
Oct 30 21:12:39.773644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.773606    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.773827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.773742    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.773827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.773784    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.773982 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.773970    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnk2h\" (UniqueName: \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-kube-api-access-qnk2h\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.774797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.774768    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.775066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.775042    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.776964 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.776920    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.786965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.786921    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnk2h\" (UniqueName: \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-kube-api-access-qnk2h\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:39.905147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.905061    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"]
Oct 30 21:12:39.930339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.920840    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:39.935206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.934201    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"]
Oct 30 21:12:39.940133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.940103    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"agc-cred-ab7d5743f570\""
Oct 30 21:12:39.955031 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.940663    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"payload-processing-external-model-e2e--d2a8fa5c-dockercfg-z42pp\""
Oct 30 21:12:39.981413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.981373    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:39.981595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.981574    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:39.981666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.981612    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5nt2\" (UniqueName: \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-kube-api-access-h5nt2\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:39.981666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:39.981646    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:40.046188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.045916    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:40.133354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.128487    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:40.133354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.130843    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5nt2\" (UniqueName: \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-kube-api-access-h5nt2\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:40.133354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.130926    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:40.133354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.131097    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:40.133354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.132612    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:40.135174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.133917    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:40.137038 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.136782    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:40.153374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.153316    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5nt2\" (UniqueName: \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-kube-api-access-h5nt2\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:40.160134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.160096    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" event={"ID":"0d459cac-0e34-4d12-91e9-ff0bfae8d20e","Type":"ContainerStarted","Data":"5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b"}
Oct 30 21:12:40.161324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.161297    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:40.163807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.163726    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g" event={"ID":"d0805f44-d79c-49c0-a247-182964ec985d","Type":"ContainerStarted","Data":"b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6"}
Oct 30 21:12:40.163807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.163783    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g" event={"ID":"d0805f44-d79c-49c0-a247-182964ec985d","Type":"ContainerStarted","Data":"29f17e0649d6cdea21f9e673507284ff9b17e827a26dbb728ad71dd8b0d4f73c"}
Oct 30 21:12:40.164052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.163839    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:40.187170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.184370    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" podStartSLOduration=3.184351286 podStartE2EDuration="3.184351286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:40.181208868 +0000 UTC m=+2126.673025412" watchObservedRunningTime="2026-10-30 21:12:40.184351286 +0000 UTC m=+2126.676167831"
Oct 30 21:12:40.213091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.213033    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g" podStartSLOduration=2.213013029 podStartE2EDuration="2.213013029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:40.209453554 +0000 UTC m=+2126.701270098" watchObservedRunningTime="2026-10-30 21:12:40.213013029 +0000 UTC m=+2126.704829573"
Oct 30 21:12:40.233481 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.233391    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d95261/agc-cred-deb2c75b9d02: secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:40.233481 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.233418    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g: secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:40.233636 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.233484    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials podName:d0805f44-d79c-49c0-a247-182964ec985d nodeName:}" failed. No retries permitted until 2026-10-30 21:12:40.733462934 +0000 UTC m=+2127.225279459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g" (UID: "d0805f44-d79c-49c0-a247-182964ec985d") : secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:40.239987 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.237947    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6741b1/agc-cred-437d372ad5d5: secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:40.239987 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.237970    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw: secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:40.239987 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.238051    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials podName:0d459cac-0e34-4d12-91e9-ff0bfae8d20e nodeName:}" failed. No retries permitted until 2026-10-30 21:12:40.738033475 +0000 UTC m=+2127.229850025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e") : secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:40.285158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.285128    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:40.295456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.294873    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"]
Oct 30 21:12:40.340733 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.339086    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-fcb5e5b64832: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:40.340733 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.339117    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:40.340733 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.339198    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials podName:d1f3dfcf-5348-4db4-a0cb-b46c6587d893 nodeName:}" failed. No retries permitted until 2026-10-30 21:12:48.339179305 +0000 UTC m=+2134.830995847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" (UID: "d1f3dfcf-5348-4db4-a0cb-b46c6587d893") : secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:40.480825 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.480784    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"]
Oct 30 21:12:40.728432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:40.728401    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"]
Oct 30 21:12:40.763446 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.762905    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d95261/agc-cred-deb2c75b9d02: secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:40.763446 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.762934    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g: secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:40.763446 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.762993    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials podName:d0805f44-d79c-49c0-a247-182964ec985d nodeName:}" failed. No retries permitted until 2026-10-30 21:12:41.762974168 +0000 UTC m=+2128.254790705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g" (UID: "d0805f44-d79c-49c0-a247-182964ec985d") : secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:40.763446 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.762913    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6741b1/agc-cred-437d372ad5d5: secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:40.763446 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.763365    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw: secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:40.763446 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:40.763417    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials podName:0d459cac-0e34-4d12-91e9-ff0bfae8d20e nodeName:}" failed. No retries permitted until 2026-10-30 21:12:41.763401133 +0000 UTC m=+2128.255217676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e") : secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:41.170844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:41.170809    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"]
Oct 30 21:12:41.174993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:41.174817    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs" event={"ID":"33df6252-a981-4979-b5e3-0cbdf4936a5a","Type":"ContainerStarted","Data":"fc176f93dfd0597000bf2611730d8a9a1882fffddec7d30cc2f785c6f0ea5507"}
Oct 30 21:12:41.174993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:41.174916    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs" event={"ID":"33df6252-a981-4979-b5e3-0cbdf4936a5a","Type":"ContainerStarted","Data":"91450b9782ce6314803e44404bcf03cd4cf97846f721dcbacd3d1eb833fd6954"}
Oct 30 21:12:41.174993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:41.174963    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:41.177689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:41.177663    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx" event={"ID":"565e83ef-38f5-4f55-bb53-14321777bf2c","Type":"ContainerStarted","Data":"e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f"}
Oct 30 21:12:41.177836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:41.177715    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx" event={"ID":"565e83ef-38f5-4f55-bb53-14321777bf2c","Type":"ContainerStarted","Data":"6f269720b60c7263b9d8c1ddcf55f161356c0ef4c8cdfef8e0861e9eff7ca83a"}
Oct 30 21:12:41.177836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:41.177745    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:41.178217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:41.178196    4810 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" secret="" err="secret \"payload-processing-external-model-e2e--a89b3f58-dockercfg-m299n\" not found"
Oct 30 21:12:41.194404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:41.194357    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs" podStartSLOduration=2.19434118 podStartE2EDuration="2.19434118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:41.193338864 +0000 UTC m=+2127.685155402" watchObservedRunningTime="2026-10-30 21:12:41.19434118 +0000 UTC m=+2127.686157727"
Oct 30 21:12:41.210032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:41.209981    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx" podStartSLOduration=2.209964903 podStartE2EDuration="2.209964903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:41.208984305 +0000 UTC m=+2127.700800861" watchObservedRunningTime="2026-10-30 21:12:41.209964903 +0000 UTC m=+2127.701781443"
Oct 30 21:12:41.268987 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:41.268957    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-plugins-e2e-extmod-a-6741b1: configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:12:41.269144 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:41.269039    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume podName:0d459cac-0e34-4d12-91e9-ff0bfae8d20e nodeName:}" failed. No retries permitted until 2026-10-30 21:12:41.769015914 +0000 UTC m=+2128.260832460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:12:41.776456 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:41.776421    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d95261/agc-cred-deb2c75b9d02: secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:41.776456 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:41.776451    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g: secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:41.776674 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:41.776466    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-plugins-e2e-extmod-a-6741b1: configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:12:41.776674 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:41.776471    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6741b1/agc-cred-437d372ad5d5: secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:41.776674 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:41.776511    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw: secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:41.776674 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:41.776516    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials podName:d0805f44-d79c-49c0-a247-182964ec985d nodeName:}" failed. No retries permitted until 2026-10-30 21:12:43.776494677 +0000 UTC m=+2130.268311203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g" (UID: "d0805f44-d79c-49c0-a247-182964ec985d") : secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:41.776674 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:41.776536    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume podName:0d459cac-0e34-4d12-91e9-ff0bfae8d20e nodeName:}" failed. No retries permitted until 2026-10-30 21:12:42.77652507 +0000 UTC m=+2129.268341593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:12:41.776674 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:41.776553    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials podName:0d459cac-0e34-4d12-91e9-ff0bfae8d20e nodeName:}" failed. No retries permitted until 2026-10-30 21:12:43.77654432 +0000 UTC m=+2130.268360845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e") : secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:42.366416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.366381    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"]
Oct 30 21:12:42.371430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.371408    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.377533 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.375268    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:42.377533 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.376216    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:12:42.379133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.379107    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"]
Oct 30 21:12:42.496325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.496287    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93319499-4c40-4c24-a456-47a018b1164e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fe7dae-4pw8c\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.496493 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.496352    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93319499-4c40-4c24-a456-47a018b1164e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fe7dae-4pw8c\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.496493 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.496424    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7dx\" (UniqueName: \"kubernetes.io/projected/93319499-4c40-4c24-a456-47a018b1164e-kube-api-access-5f7dx\") pod \"maas-api-revoke-sub-70fe7dae-4pw8c\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.598627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.598587    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93319499-4c40-4c24-a456-47a018b1164e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fe7dae-4pw8c\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.598794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.598650    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93319499-4c40-4c24-a456-47a018b1164e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fe7dae-4pw8c\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.598794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.598685    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7dx\" (UniqueName: \"kubernetes.io/projected/93319499-4c40-4c24-a456-47a018b1164e-kube-api-access-5f7dx\") pod \"maas-api-revoke-sub-70fe7dae-4pw8c\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.599418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.599391    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93319499-4c40-4c24-a456-47a018b1164e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fe7dae-4pw8c\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.601249 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.601222    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93319499-4c40-4c24-a456-47a018b1164e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fe7dae-4pw8c\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.607977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.607952    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7dx\" (UniqueName: \"kubernetes.io/projected/93319499-4c40-4c24-a456-47a018b1164e-kube-api-access-5f7dx\") pod \"maas-api-revoke-sub-70fe7dae-4pw8c\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.648273 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.648249    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:42.698424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.698088    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:42.819322 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:42.819287    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-plugins-e2e-extmod-a-6741b1: configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:12:42.819482 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:42.819368    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume podName:0d459cac-0e34-4d12-91e9-ff0bfae8d20e nodeName:}" failed. No retries permitted until 2026-10-30 21:12:44.819349012 +0000 UTC m=+2131.311165558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:12:42.990130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:42.990096    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"]
Oct 30 21:12:42.990664 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:12:42.990633    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93319499_4c40_4c24_a456_47a018b1164e.slice/crio-0ded46f0807db7463d6ab7b7ea0dac6a68f6734972f92f585a5a4f3a3626ccdc WatchSource:0}: Error finding container 0ded46f0807db7463d6ab7b7ea0dac6a68f6734972f92f585a5a4f3a3626ccdc: Status 404 returned error can't find the container with id 0ded46f0807db7463d6ab7b7ea0dac6a68f6734972f92f585a5a4f3a3626ccdc
Oct 30 21:12:43.201012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:43.200976    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c" event={"ID":"93319499-4c40-4c24-a456-47a018b1164e","Type":"ContainerStarted","Data":"0ded46f0807db7463d6ab7b7ea0dac6a68f6734972f92f585a5a4f3a3626ccdc"}
Oct 30 21:12:43.836045 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:43.835967    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6741b1/agc-cred-437d372ad5d5: secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:43.836045 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:43.835998    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw: secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:43.836453 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:43.836064    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials podName:0d459cac-0e34-4d12-91e9-ff0bfae8d20e nodeName:}" failed. No retries permitted until 2026-10-30 21:12:47.836044919 +0000 UTC m=+2134.327861494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e") : secret "agc-cred-437d372ad5d5" not found
Oct 30 21:12:43.836453 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:43.836076    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d95261/agc-cred-deb2c75b9d02: secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:43.836453 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:43.836094    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g: secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:43.836453 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:43.836138    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials podName:d0805f44-d79c-49c0-a247-182964ec985d nodeName:}" failed. No retries permitted until 2026-10-30 21:12:47.836125625 +0000 UTC m=+2134.327942161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g" (UID: "d0805f44-d79c-49c0-a247-182964ec985d") : secret "agc-cred-deb2c75b9d02" not found
Oct 30 21:12:44.212632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:44.212594    4810 generic.go:356] "Generic (PLEG): container finished" podID="93319499-4c40-4c24-a456-47a018b1164e" containerID="2a40e75d384774339bfd4377389b3fab2bc027d05fbfc408fc6f08cdf9fea246" exitCode=0
Oct 30 21:12:44.212804 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:44.212648    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c" event={"ID":"93319499-4c40-4c24-a456-47a018b1164e","Type":"ContainerDied","Data":"2a40e75d384774339bfd4377389b3fab2bc027d05fbfc408fc6f08cdf9fea246"}
Oct 30 21:12:44.724908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:44.724847    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"]
Oct 30 21:12:44.856197 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:44.856160    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-plugins-e2e-extmod-a-6741b1: configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:12:44.856574 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:44.856237    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume podName:0d459cac-0e34-4d12-91e9-ff0bfae8d20e nodeName:}" failed. No retries permitted until 2026-10-30 21:12:48.856218399 +0000 UTC m=+2135.348034922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:12:45.402392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.402369    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:45.466416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.466379    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93319499-4c40-4c24-a456-47a018b1164e-kube-api-access-5f7dx\" (UniqueName: \"kubernetes.io/projected/93319499-4c40-4c24-a456-47a018b1164e-kube-api-access-5f7dx\") pod \"93319499-4c40-4c24-a456-47a018b1164e\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") "
Oct 30 21:12:45.466588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.466546    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93319499-4c40-4c24-a456-47a018b1164e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93319499-4c40-4c24-a456-47a018b1164e-maas-api-openshift-ca\") pod \"93319499-4c40-4c24-a456-47a018b1164e\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") "
Oct 30 21:12:45.467143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.466709    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93319499-4c40-4c24-a456-47a018b1164e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93319499-4c40-4c24-a456-47a018b1164e-maas-api-xks-ca\") pod \"93319499-4c40-4c24-a456-47a018b1164e\" (UID: \"93319499-4c40-4c24-a456-47a018b1164e\") "
Oct 30 21:12:45.467143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.466877    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93319499-4c40-4c24-a456-47a018b1164e-maas-api-openshift-ca" pod "93319499-4c40-4c24-a456-47a018b1164e" (UID: "93319499-4c40-4c24-a456-47a018b1164e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:45.467579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.467555    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93319499-4c40-4c24-a456-47a018b1164e-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:45.468831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.468807    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93319499-4c40-4c24-a456-47a018b1164e-kube-api-access-5f7dx" pod "93319499-4c40-4c24-a456-47a018b1164e" (UID: "93319499-4c40-4c24-a456-47a018b1164e"). InnerVolumeSpecName "kube-api-access-5f7dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:45.469358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.469336    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93319499-4c40-4c24-a456-47a018b1164e-maas-api-xks-ca" pod "93319499-4c40-4c24-a456-47a018b1164e" (UID: "93319499-4c40-4c24-a456-47a018b1164e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:45.569040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.569004    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93319499-4c40-4c24-a456-47a018b1164e-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:45.569040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.569034    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7dx\" (UniqueName: \"kubernetes.io/projected/93319499-4c40-4c24-a456-47a018b1164e-kube-api-access-5f7dx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:45.773050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.772977    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:12:45.826778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:45.826743    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"]
Oct 30 21:12:46.233123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:46.233092    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"
Oct 30 21:12:46.233548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:46.233134    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c" event={"ID":"93319499-4c40-4c24-a456-47a018b1164e","Type":"ContainerDied","Data":"0ded46f0807db7463d6ab7b7ea0dac6a68f6734972f92f585a5a4f3a3626ccdc"}
Oct 30 21:12:46.233548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:46.233194    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ded46f0807db7463d6ab7b7ea0dac6a68f6734972f92f585a5a4f3a3626ccdc"
Oct 30 21:12:46.824807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:46.824777    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:12:47.029745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.029709    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"]
Oct 30 21:12:47.038100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.038071    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f91ecb99-mvftp"]
Oct 30 21:12:47.188947 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.188908    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" podUID="0d459cac-0e34-4d12-91e9-ff0bfae8d20e" containerName="payload-processing" containerID="cri-o://5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b" gracePeriod=25
Oct 30 21:12:47.189117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.188926    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g" podUID="d0805f44-d79c-49c0-a247-182964ec985d" containerName="payload-processing" containerID="cri-o://b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6" gracePeriod=25
Oct 30 21:12:47.233361 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:47.233329    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0805f44_d79c_49c0_a247_182964ec985d.slice/crio-b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:12:47.509519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.509487    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:47.531903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.531844    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:47.607103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.607066    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials\") pod \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") "
Oct 30 21:12:47.607319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.607120    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-routing-overlay-volume\") pod \"d0805f44-d79c-49c0-a247-182964ec985d\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") "
Oct 30 21:12:47.607428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.607403    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume\") pod \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") "
Oct 30 21:12:47.607492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.607453    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials\") pod \"d0805f44-d79c-49c0-a247-182964ec985d\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") "
Oct 30 21:12:47.607544 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.607505    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-routing-overlay-volume\") pod \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") "
Oct 30 21:12:47.607595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.607554    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-kube-api-access-5hhk2\" (UniqueName: \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-kube-api-access-5hhk2\") pod \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\" (UID: \"0d459cac-0e34-4d12-91e9-ff0bfae8d20e\") "
Oct 30 21:12:47.607649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.607640    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-plugins-config-volume\") pod \"d0805f44-d79c-49c0-a247-182964ec985d\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") "
Oct 30 21:12:47.607728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.607710    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-kube-api-access-wd869\" (UniqueName: \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-kube-api-access-wd869\") pod \"d0805f44-d79c-49c0-a247-182964ec985d\" (UID: \"d0805f44-d79c-49c0-a247-182964ec985d\") "
Oct 30 21:12:47.608783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.607939    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume" pod "0d459cac-0e34-4d12-91e9-ff0bfae8d20e" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:47.608783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.608610    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:47.608783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.608680    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-routing-overlay-volume" pod "0d459cac-0e34-4d12-91e9-ff0bfae8d20e" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:47.608783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.608709    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-plugins-config-volume" pod "d0805f44-d79c-49c0-a247-182964ec985d" (UID: "d0805f44-d79c-49c0-a247-182964ec985d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:47.609649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.609627    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-routing-overlay-volume" pod "d0805f44-d79c-49c0-a247-182964ec985d" (UID: "d0805f44-d79c-49c0-a247-182964ec985d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:47.615338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.615294    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-kube-api-access-5hhk2" pod "0d459cac-0e34-4d12-91e9-ff0bfae8d20e" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e"). InnerVolumeSpecName "kube-api-access-5hhk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:47.615924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.615836    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-kube-api-access-wd869" pod "d0805f44-d79c-49c0-a247-182964ec985d" (UID: "d0805f44-d79c-49c0-a247-182964ec985d"). InnerVolumeSpecName "kube-api-access-wd869". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:47.617050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.617024    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials" pod "0d459cac-0e34-4d12-91e9-ff0bfae8d20e" (UID: "0d459cac-0e34-4d12-91e9-ff0bfae8d20e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:47.618740 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.618708    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials" pod "d0805f44-d79c-49c0-a247-182964ec985d" (UID: "d0805f44-d79c-49c0-a247-182964ec985d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:47.664952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.664918    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b55e2fea-36c7-4d70-8962-bfc58ea292ed" path="/var/lib/kubelet/pods/b55e2fea-36c7-4d70-8962-bfc58ea292ed/volumes"
Oct 30 21:12:47.710679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.710605    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:47.710881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.710845    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:47.710967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.710886    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hhk2\" (UniqueName: \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-kube-api-access-5hhk2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:47.710967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.710901    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:47.710967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.710915    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd869\" (UniqueName: \"kubernetes.io/projected/d0805f44-d79c-49c0-a247-182964ec985d-kube-api-access-wd869\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:47.710967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.710930    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d459cac-0e34-4d12-91e9-ff0bfae8d20e-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:47.710967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:47.710944    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0805f44-d79c-49c0-a247-182964ec985d-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:48.250704 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.250667    4810 generic.go:356] "Generic (PLEG): container finished" podID="d0805f44-d79c-49c0-a247-182964ec985d" containerID="b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6" exitCode=0
Oct 30 21:12:48.251193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.250719    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"
Oct 30 21:12:48.251193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.250760    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g" event={"ID":"d0805f44-d79c-49c0-a247-182964ec985d","Type":"ContainerDied","Data":"b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6"}
Oct 30 21:12:48.251193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.250835    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g" event={"ID":"d0805f44-d79c-49c0-a247-182964ec985d","Type":"ContainerDied","Data":"29f17e0649d6cdea21f9e673507284ff9b17e827a26dbb728ad71dd8b0d4f73c"}
Oct 30 21:12:48.251193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.250881    4810 scope.go:122] "RemoveContainer" containerID="b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6"
Oct 30 21:12:48.252826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.252800    4810 generic.go:356] "Generic (PLEG): container finished" podID="0d459cac-0e34-4d12-91e9-ff0bfae8d20e" containerID="5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b" exitCode=0
Oct 30 21:12:48.253020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.252919    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"
Oct 30 21:12:48.253439 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.253408    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" event={"ID":"0d459cac-0e34-4d12-91e9-ff0bfae8d20e","Type":"ContainerDied","Data":"5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b"}
Oct 30 21:12:48.253570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.253470    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw" event={"ID":"0d459cac-0e34-4d12-91e9-ff0bfae8d20e","Type":"ContainerDied","Data":"d94672ccd77ffd0b481982b4692135bac0424d835b43f2cbd8d9276a4d42404e"}
Oct 30 21:12:48.263384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.263365    4810 scope.go:122] "RemoveContainer" containerID="b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6"
Oct 30 21:12:48.263736 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:48.263712    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6\": container with ID starting with b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6 not found: ID does not exist" containerID="b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6"
Oct 30 21:12:48.263831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.263743    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6"} err="failed to get container status \"b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6\": rpc error: code = NotFound desc = could not find container \"b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6\": container with ID starting with b03bdebb97f9600946d664fa0c0b9b791bee45275fdfd525b43650919839b1b6 not found: ID does not exist"
Oct 30 21:12:48.263831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.263758    4810 scope.go:122] "RemoveContainer" containerID="5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b"
Oct 30 21:12:48.274364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.274275    4810 scope.go:122] "RemoveContainer" containerID="5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b"
Oct 30 21:12:48.274924 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:48.274808    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b\": container with ID starting with 5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b not found: ID does not exist" containerID="5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b"
Oct 30 21:12:48.274924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.274845    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b"} err="failed to get container status \"5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b\": rpc error: code = NotFound desc = could not find container \"5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b\": container with ID starting with 5350624d2ad3cd5bb57d24a847d808a5383f195b923f0eef08794965db6c3d7b not found: ID does not exist"
Oct 30 21:12:48.282185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.282157    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"]
Oct 30 21:12:48.289583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.289552    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-b9cddnf5g"]
Oct 30 21:12:48.299966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.299934    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"]
Oct 30 21:12:48.304534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:48.304507    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-7c77t6rvw"]
Oct 30 21:12:48.422356 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:48.422324    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-fcb5e5b64832: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:48.422356 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:48.422351    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:48.422565 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:48.422411    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials podName:d1f3dfcf-5348-4db4-a0cb-b46c6587d893 nodeName:}" failed. No retries permitted until 2026-10-30 21:13:04.422395188 +0000 UTC m=+2150.914211711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" (UID: "d1f3dfcf-5348-4db4-a0cb-b46c6587d893") : secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:12:49.665477 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:49.665441    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d459cac-0e34-4d12-91e9-ff0bfae8d20e" path="/var/lib/kubelet/pods/0d459cac-0e34-4d12-91e9-ff0bfae8d20e/volumes"
Oct 30 21:12:49.665984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:49.665886    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0805f44-d79c-49c0-a247-182964ec985d" path="/var/lib/kubelet/pods/d0805f44-d79c-49c0-a247-182964ec985d/volumes"
Oct 30 21:12:49.726212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:49.726171    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx" podUID="565e83ef-38f5-4f55-bb53-14321777bf2c" containerName="payload-processing" containerID="cri-o://e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f" gracePeriod=25
Oct 30 21:12:50.028385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.028352    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:50.145320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.145267    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-plugins-config-volume\") pod \"565e83ef-38f5-4f55-bb53-14321777bf2c\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") "
Oct 30 21:12:50.145504 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.145405    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-provider-credentials\") pod \"565e83ef-38f5-4f55-bb53-14321777bf2c\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") "
Oct 30 21:12:50.145683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.145666    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-kube-api-access-qnk2h\" (UniqueName: \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-kube-api-access-qnk2h\") pod \"565e83ef-38f5-4f55-bb53-14321777bf2c\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") "
Oct 30 21:12:50.145818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.145803    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-routing-overlay-volume\") pod \"565e83ef-38f5-4f55-bb53-14321777bf2c\" (UID: \"565e83ef-38f5-4f55-bb53-14321777bf2c\") "
Oct 30 21:12:50.146893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.146846    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-routing-overlay-volume" pod "565e83ef-38f5-4f55-bb53-14321777bf2c" (UID: "565e83ef-38f5-4f55-bb53-14321777bf2c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:50.147017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.146976    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-plugins-config-volume" pod "565e83ef-38f5-4f55-bb53-14321777bf2c" (UID: "565e83ef-38f5-4f55-bb53-14321777bf2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:50.148959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.148929    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-provider-credentials" pod "565e83ef-38f5-4f55-bb53-14321777bf2c" (UID: "565e83ef-38f5-4f55-bb53-14321777bf2c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:50.149090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.149059    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-kube-api-access-qnk2h" pod "565e83ef-38f5-4f55-bb53-14321777bf2c" (UID: "565e83ef-38f5-4f55-bb53-14321777bf2c"). InnerVolumeSpecName "kube-api-access-qnk2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:50.247657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.247578    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:50.247657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.247608    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnk2h\" (UniqueName: \"kubernetes.io/projected/565e83ef-38f5-4f55-bb53-14321777bf2c-kube-api-access-qnk2h\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:50.247657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.247618    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:50.247657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.247628    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/565e83ef-38f5-4f55-bb53-14321777bf2c-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:50.273408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.273374    4810 generic.go:356] "Generic (PLEG): container finished" podID="565e83ef-38f5-4f55-bb53-14321777bf2c" containerID="e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f" exitCode=0
Oct 30 21:12:50.273574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.273415    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"
Oct 30 21:12:50.273574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.273459    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx" event={"ID":"565e83ef-38f5-4f55-bb53-14321777bf2c","Type":"ContainerDied","Data":"e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f"}
Oct 30 21:12:50.273574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.273521    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx" event={"ID":"565e83ef-38f5-4f55-bb53-14321777bf2c","Type":"ContainerDied","Data":"6f269720b60c7263b9d8c1ddcf55f161356c0ef4c8cdfef8e0861e9eff7ca83a"}
Oct 30 21:12:50.273574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.273552    4810 scope.go:122] "RemoveContainer" containerID="e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f"
Oct 30 21:12:50.286896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.286851    4810 scope.go:122] "RemoveContainer" containerID="e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f"
Oct 30 21:12:50.287222 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:50.287189    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f\": container with ID starting with e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f not found: ID does not exist" containerID="e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f"
Oct 30 21:12:50.287333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.287224    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f"} err="failed to get container status \"e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f\": rpc error: code = NotFound desc = could not find container \"e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f\": container with ID starting with e04dedeb35ab1035500dfd41c86fd1f7d0a8eccea19b3974df2790947192487f not found: ID does not exist"
Oct 30 21:12:50.303901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.303871    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"]
Oct 30 21:12:50.308916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.308885    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-c884mpgjx"]
Oct 30 21:12:50.831883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:50.828254    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" podUID="d1f3dfcf-5348-4db4-a0cb-b46c6587d893" containerName="payload-processing" containerID="cri-o://4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861" gracePeriod=25
Oct 30 21:12:51.123443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.123418    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:51.263703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.263670    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-routing-overlay-volume\") pod \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") "
Oct 30 21:12:51.263981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.263793    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-kube-api-access-bhcxl\" (UniqueName: \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-kube-api-access-bhcxl\") pod \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") "
Oct 30 21:12:51.264242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.264206    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials\") pod \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") "
Oct 30 21:12:51.264339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.264287    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-plugins-config-volume\") pod \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\" (UID: \"d1f3dfcf-5348-4db4-a0cb-b46c6587d893\") "
Oct 30 21:12:51.265079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.265051    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-routing-overlay-volume" pod "d1f3dfcf-5348-4db4-a0cb-b46c6587d893" (UID: "d1f3dfcf-5348-4db4-a0cb-b46c6587d893"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:51.265426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.265341    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-plugins-config-volume" pod "d1f3dfcf-5348-4db4-a0cb-b46c6587d893" (UID: "d1f3dfcf-5348-4db4-a0cb-b46c6587d893"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:51.267504 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.267478    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials" pod "d1f3dfcf-5348-4db4-a0cb-b46c6587d893" (UID: "d1f3dfcf-5348-4db4-a0cb-b46c6587d893"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:51.268898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.268541    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-kube-api-access-bhcxl" pod "d1f3dfcf-5348-4db4-a0cb-b46c6587d893" (UID: "d1f3dfcf-5348-4db4-a0cb-b46c6587d893"). InnerVolumeSpecName "kube-api-access-bhcxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:51.287424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.287094    4810 generic.go:356] "Generic (PLEG): container finished" podID="d1f3dfcf-5348-4db4-a0cb-b46c6587d893" containerID="4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861" exitCode=0
Oct 30 21:12:51.287424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.287180    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" event={"ID":"d1f3dfcf-5348-4db4-a0cb-b46c6587d893","Type":"ContainerDied","Data":"4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861"}
Oct 30 21:12:51.287424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.287228    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94" event={"ID":"d1f3dfcf-5348-4db4-a0cb-b46c6587d893","Type":"ContainerDied","Data":"1c8ec9aead3f202b0da2829cdd0c805a3c622d22caa6bd396778a452cfa952c5"}
Oct 30 21:12:51.287424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.287258    4810 scope.go:122] "RemoveContainer" containerID="4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861"
Oct 30 21:12:51.287424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.287407    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"
Oct 30 21:12:51.307745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.307719    4810 scope.go:122] "RemoveContainer" containerID="4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861"
Oct 30 21:12:51.308143 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:51.308122    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861\": container with ID starting with 4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861 not found: ID does not exist" containerID="4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861"
Oct 30 21:12:51.308222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.308156    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861"} err="failed to get container status \"4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861\": rpc error: code = NotFound desc = could not find container \"4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861\": container with ID starting with 4274912a21ba529dc855beaf006d46691555f15c3ea62f1315105510af7f1861 not found: ID does not exist"
Oct 30 21:12:51.317247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.317221    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"]
Oct 30 21:12:51.323600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.323573    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fcbd94"]
Oct 30 21:12:51.366803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.366748    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:51.366803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.366774    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhcxl\" (UniqueName: \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-kube-api-access-bhcxl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:51.366803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.366785    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:51.366803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.366794    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f3dfcf-5348-4db4-a0cb-b46c6587d893-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:51.665649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.665617    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565e83ef-38f5-4f55-bb53-14321777bf2c" path="/var/lib/kubelet/pods/565e83ef-38f5-4f55-bb53-14321777bf2c/volumes"
Oct 30 21:12:51.666163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:51.666148    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f3dfcf-5348-4db4-a0cb-b46c6587d893" path="/var/lib/kubelet/pods/d1f3dfcf-5348-4db4-a0cb-b46c6587d893/volumes"
Oct 30 21:12:52.192637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:52.192606    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:12:52.245622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:52.245588    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"]
Oct 30 21:12:53.561305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.561266    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"]
Oct 30 21:12:53.562210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562186    4810 cpu_manager.go:414] "containerMap: removing container" podUID="565e83ef-38f5-4f55-bb53-14321777bf2c" containerName="payload-processing"
Oct 30 21:12:53.562412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562388    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565e83ef-38f5-4f55-bb53-14321777bf2c" containerName="payload-processing"
Oct 30 21:12:53.562523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562458    4810 cpu_manager.go:414] "containerMap: removing container" podUID="93319499-4c40-4c24-a456-47a018b1164e" containerName="revoke-keys"
Oct 30 21:12:53.562523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562467    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93319499-4c40-4c24-a456-47a018b1164e" containerName="revoke-keys"
Oct 30 21:12:53.562523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562486    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d0805f44-d79c-49c0-a247-182964ec985d" containerName="payload-processing"
Oct 30 21:12:53.562523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562493    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0805f44-d79c-49c0-a247-182964ec985d" containerName="payload-processing"
Oct 30 21:12:53.562523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562514    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d1f3dfcf-5348-4db4-a0cb-b46c6587d893" containerName="payload-processing"
Oct 30 21:12:53.562523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562523    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f3dfcf-5348-4db4-a0cb-b46c6587d893" containerName="payload-processing"
Oct 30 21:12:53.562733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562563    4810 cpu_manager.go:414] "containerMap: removing container" podUID="0d459cac-0e34-4d12-91e9-ff0bfae8d20e" containerName="payload-processing"
Oct 30 21:12:53.562733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562570    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d459cac-0e34-4d12-91e9-ff0bfae8d20e" containerName="payload-processing"
Oct 30 21:12:53.562794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562778    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="93319499-4c40-4c24-a456-47a018b1164e" containerName="revoke-keys"
Oct 30 21:12:53.562830 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562795    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0805f44-d79c-49c0-a247-182964ec985d" containerName="payload-processing"
Oct 30 21:12:53.562830 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562808    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f3dfcf-5348-4db4-a0cb-b46c6587d893" containerName="payload-processing"
Oct 30 21:12:53.562830 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562824    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d459cac-0e34-4d12-91e9-ff0bfae8d20e" containerName="payload-processing"
Oct 30 21:12:53.562975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.562835    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="565e83ef-38f5-4f55-bb53-14321777bf2c" containerName="payload-processing"
Oct 30 21:12:53.568817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.568785    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.574629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.574421    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"]
Oct 30 21:12:53.612116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.612077    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.612448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.612172    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.612448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.612246    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.612448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.612333    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88z8\" (UniqueName: \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-kube-api-access-f88z8\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.713504 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.713465    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88z8\" (UniqueName: \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-kube-api-access-f88z8\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.713723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.713704    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.713822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.713749    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.713822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.713813    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.714484 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.714461    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.714998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.714973    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.716660 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.716632    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.726015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.725988    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88z8\" (UniqueName: \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-kube-api-access-f88z8\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:53.894879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:53.890954    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:54.111232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.106892    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"]
Oct 30 21:12:54.135213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.134628    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.135594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.135557    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"]
Oct 30 21:12:54.140949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.139001    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:12:54.140949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.139345    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:54.235915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.235874    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf115403-d57mv\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.236090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.235921    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf115403-d57mv\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.236090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.236079    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwmt7\" (UniqueName: \"kubernetes.io/projected/9896c083-00b2-425b-a9c4-14f79bd66cac-kube-api-access-jwmt7\") pod \"maas-api-revoke-sub-bf115403-d57mv\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.316141 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.316112    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"]
Oct 30 21:12:54.316739 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:12:54.316710    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda059db35_6bd6_47aa_a491_efe1f48606e1.slice/crio-fc07b29ea8963441fe4852626ef6b64cebda454ae4b4ee81a3181103e5f07e60 WatchSource:0}: Error finding container fc07b29ea8963441fe4852626ef6b64cebda454ae4b4ee81a3181103e5f07e60: Status 404 returned error can't find the container with id fc07b29ea8963441fe4852626ef6b64cebda454ae4b4ee81a3181103e5f07e60
Oct 30 21:12:54.329952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.329921    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw" event={"ID":"a059db35-6bd6-47aa-a491-efe1f48606e1","Type":"ContainerStarted","Data":"fc07b29ea8963441fe4852626ef6b64cebda454ae4b4ee81a3181103e5f07e60"}
Oct 30 21:12:54.337914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.337892    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf115403-d57mv\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.337984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.337938    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf115403-d57mv\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.338115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.338097    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmt7\" (UniqueName: \"kubernetes.io/projected/9896c083-00b2-425b-a9c4-14f79bd66cac-kube-api-access-jwmt7\") pod \"maas-api-revoke-sub-bf115403-d57mv\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.338574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.338549    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf115403-d57mv\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.340365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.340340    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf115403-d57mv\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.346987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.346963    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmt7\" (UniqueName: \"kubernetes.io/projected/9896c083-00b2-425b-a9c4-14f79bd66cac-kube-api-access-jwmt7\") pod \"maas-api-revoke-sub-bf115403-d57mv\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.457289 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.457229    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:54.743961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:54.743917    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"]
Oct 30 21:12:54.745490 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:12:54.745453    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9896c083_00b2_425b_a9c4_14f79bd66cac.slice/crio-c6393dad362d649f5ce41b4c45225ae7bac95c975e760dfd44b80cba6736a332 WatchSource:0}: Error finding container c6393dad362d649f5ce41b4c45225ae7bac95c975e760dfd44b80cba6736a332: Status 404 returned error can't find the container with id c6393dad362d649f5ce41b4c45225ae7bac95c975e760dfd44b80cba6736a332
Oct 30 21:12:55.157632 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:12:55.157601    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:12:55.345156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:55.345119    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv" event={"ID":"9896c083-00b2-425b-a9c4-14f79bd66cac","Type":"ContainerStarted","Data":"63b56814691fa6d87f0ff3b7a06604441ee2f9ae2cb14f56d2cccddb93f46fb3"}
Oct 30 21:12:55.345356 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:55.345186    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv" event={"ID":"9896c083-00b2-425b-a9c4-14f79bd66cac","Type":"ContainerStarted","Data":"c6393dad362d649f5ce41b4c45225ae7bac95c975e760dfd44b80cba6736a332"}
Oct 30 21:12:55.346813 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:55.346784    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw" event={"ID":"a059db35-6bd6-47aa-a491-efe1f48606e1","Type":"ContainerStarted","Data":"cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1"}
Oct 30 21:12:55.346940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:55.346833    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:12:55.375687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:55.375642    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv" podStartSLOduration=1.375627371 podStartE2EDuration="1.375627371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:55.372392433 +0000 UTC m=+2141.864208990" watchObservedRunningTime="2026-10-30 21:12:55.375627371 +0000 UTC m=+2141.867443920"
Oct 30 21:12:55.401469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:55.401424    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw" podStartSLOduration=2.401410855 podStartE2EDuration="2.401410855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:55.397987566 +0000 UTC m=+2141.889804112" watchObservedRunningTime="2026-10-30 21:12:55.401410855 +0000 UTC m=+2141.893227394"
Oct 30 21:12:56.356404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:56.356374    4810 generic.go:356] "Generic (PLEG): container finished" podID="9896c083-00b2-425b-a9c4-14f79bd66cac" containerID="63b56814691fa6d87f0ff3b7a06604441ee2f9ae2cb14f56d2cccddb93f46fb3" exitCode=0
Oct 30 21:12:56.356838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:56.356460    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv" event={"ID":"9896c083-00b2-425b-a9c4-14f79bd66cac","Type":"ContainerDied","Data":"63b56814691fa6d87f0ff3b7a06604441ee2f9ae2cb14f56d2cccddb93f46fb3"}
Oct 30 21:12:57.246543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.246501    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz" podUID="cbf83ddb-6619-4e59-a82c-dc90d9eb0f02" containerName="payload-processing" containerID="cri-o://244ed0a39ffb094205b5d90ba3fee8acca34649f2d9a6c2a98c921cb60dacc99" gracePeriod=25
Oct 30 21:12:57.366517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.366488    4810 generic.go:356] "Generic (PLEG): container finished" podID="cbf83ddb-6619-4e59-a82c-dc90d9eb0f02" containerID="244ed0a39ffb094205b5d90ba3fee8acca34649f2d9a6c2a98c921cb60dacc99" exitCode=0
Oct 30 21:12:57.366842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.366556    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz" event={"ID":"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02","Type":"ContainerDied","Data":"244ed0a39ffb094205b5d90ba3fee8acca34649f2d9a6c2a98c921cb60dacc99"}
Oct 30 21:12:57.582578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.582552    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:57.587137 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.587113    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:12:57.698837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.698800    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-provider-credentials\") pod \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") "
Oct 30 21:12:57.699049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.698938    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-plugins-config-volume\") pod \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") "
Oct 30 21:12:57.699049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.698988    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-kube-api-access-9dffs\" (UniqueName: \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-kube-api-access-9dffs\") pod \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") "
Oct 30 21:12:57.699049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.699025    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-xks-ca\") pod \"9896c083-00b2-425b-a9c4-14f79bd66cac\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") "
Oct 30 21:12:57.699221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.699159    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-routing-overlay-volume\") pod \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\" (UID: \"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02\") "
Oct 30 21:12:57.699221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.699195    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9896c083-00b2-425b-a9c4-14f79bd66cac-kube-api-access-jwmt7\" (UniqueName: \"kubernetes.io/projected/9896c083-00b2-425b-a9c4-14f79bd66cac-kube-api-access-jwmt7\") pod \"9896c083-00b2-425b-a9c4-14f79bd66cac\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") "
Oct 30 21:12:57.699334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.699246    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-openshift-ca\") pod \"9896c083-00b2-425b-a9c4-14f79bd66cac\" (UID: \"9896c083-00b2-425b-a9c4-14f79bd66cac\") "
Oct 30 21:12:57.699334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.699307    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-plugins-config-volume" pod "cbf83ddb-6619-4e59-a82c-dc90d9eb0f02" (UID: "cbf83ddb-6619-4e59-a82c-dc90d9eb0f02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:57.699824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.699701    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-routing-overlay-volume" pod "cbf83ddb-6619-4e59-a82c-dc90d9eb0f02" (UID: "cbf83ddb-6619-4e59-a82c-dc90d9eb0f02"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:57.700036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.699987    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-openshift-ca" pod "9896c083-00b2-425b-a9c4-14f79bd66cac" (UID: "9896c083-00b2-425b-a9c4-14f79bd66cac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:57.700100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.700054    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:57.700100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.700080    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:57.701805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.701703    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-provider-credentials" pod "cbf83ddb-6619-4e59-a82c-dc90d9eb0f02" (UID: "cbf83ddb-6619-4e59-a82c-dc90d9eb0f02"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:57.701941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.701843    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-kube-api-access-9dffs" pod "cbf83ddb-6619-4e59-a82c-dc90d9eb0f02" (UID: "cbf83ddb-6619-4e59-a82c-dc90d9eb0f02"). InnerVolumeSpecName "kube-api-access-9dffs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:57.702103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.702072    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-xks-ca" pod "9896c083-00b2-425b-a9c4-14f79bd66cac" (UID: "9896c083-00b2-425b-a9c4-14f79bd66cac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:57.702208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.702158    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9896c083-00b2-425b-a9c4-14f79bd66cac-kube-api-access-jwmt7" pod "9896c083-00b2-425b-a9c4-14f79bd66cac" (UID: "9896c083-00b2-425b-a9c4-14f79bd66cac"). InnerVolumeSpecName "kube-api-access-jwmt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:57.802201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.802105    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwmt7\" (UniqueName: \"kubernetes.io/projected/9896c083-00b2-425b-a9c4-14f79bd66cac-kube-api-access-jwmt7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:57.802201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.802141    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:57.802201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.802156    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:57.802201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.802168    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dffs\" (UniqueName: \"kubernetes.io/projected/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02-kube-api-access-9dffs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:57.802201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:57.802183    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9896c083-00b2-425b-a9c4-14f79bd66cac-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:12:58.030179 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:58.030144    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"]
Oct 30 21:12:58.043623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:58.043586    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b06f71-46jh2"]
Oct 30 21:12:58.376965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:58.376931    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"
Oct 30 21:12:58.376965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:58.376955    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz" event={"ID":"cbf83ddb-6619-4e59-a82c-dc90d9eb0f02","Type":"ContainerDied","Data":"e173471e6ca383d7f31977667ab2bff99cc378f7a156793b22687459a0426fe5"}
Oct 30 21:12:58.377504 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:58.377032    4810 scope.go:122] "RemoveContainer" containerID="244ed0a39ffb094205b5d90ba3fee8acca34649f2d9a6c2a98c921cb60dacc99"
Oct 30 21:12:58.378832 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:58.378808    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv" event={"ID":"9896c083-00b2-425b-a9c4-14f79bd66cac","Type":"ContainerDied","Data":"c6393dad362d649f5ce41b4c45225ae7bac95c975e760dfd44b80cba6736a332"}
Oct 30 21:12:58.378991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:58.378849    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6393dad362d649f5ce41b4c45225ae7bac95c975e760dfd44b80cba6736a332"
Oct 30 21:12:58.378991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:58.378980    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"
Oct 30 21:12:58.410592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:58.410562    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"]
Oct 30 21:12:58.417653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:58.417628    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb6nm5mz"]
Oct 30 21:12:59.664819 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:59.664787    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf83ddb-6619-4e59-a82c-dc90d9eb0f02" path="/var/lib/kubelet/pods/cbf83ddb-6619-4e59-a82c-dc90d9eb0f02/volumes"
Oct 30 21:12:59.665320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:12:59.665303    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a9430a-1874-4a1f-b118-f61e2d57f202" path="/var/lib/kubelet/pods/d8a9430a-1874-4a1f-b118-f61e2d57f202/volumes"
Oct 30 21:13:05.238895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:05.232946    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"]
Oct 30 21:13:05.238895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:05.236614    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"]
Oct 30 21:13:05.249878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:05.244964    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:13:09.449949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.449908    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"]
Oct 30 21:13:09.452895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.451807    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9896c083-00b2-425b-a9c4-14f79bd66cac" containerName="revoke-keys"
Oct 30 21:13:09.453123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.453098    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9896c083-00b2-425b-a9c4-14f79bd66cac" containerName="revoke-keys"
Oct 30 21:13:09.453242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.453231    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cbf83ddb-6619-4e59-a82c-dc90d9eb0f02" containerName="payload-processing"
Oct 30 21:13:09.453324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.453313    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf83ddb-6619-4e59-a82c-dc90d9eb0f02" containerName="payload-processing"
Oct 30 21:13:09.453742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.453708    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9896c083-00b2-425b-a9c4-14f79bd66cac" containerName="revoke-keys"
Oct 30 21:13:09.453922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.453906    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf83ddb-6619-4e59-a82c-dc90d9eb0f02" containerName="payload-processing"
Oct 30 21:13:09.471192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.471164    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.506604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.503599    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"]
Oct 30 21:13:09.617932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.605801    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw4zm\" (UniqueName: \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-kube-api-access-dw4zm\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.617932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.605921    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.617932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.606121    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.617932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.606157    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.707911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.707786    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.707911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.707837    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.708185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.708047    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4zm\" (UniqueName: \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-kube-api-access-dw4zm\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.708185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.708118    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.708602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.708579    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.709016 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.708994    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.710492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.710471    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.717229 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.717205    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4zm\" (UniqueName: \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-kube-api-access-dw4zm\") pod \"payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") " pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:09.896550 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:09.896518    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:10.193901 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:13:10.191138    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0e9c8b_42b5_4258_ab33_d91a39d8c3cb.slice/crio-140e1a7f0a215acba735d91f6a862910ebfd304a819860d0e50f33f515e34c9b WatchSource:0}: Error finding container 140e1a7f0a215acba735d91f6a862910ebfd304a819860d0e50f33f515e34c9b: Status 404 returned error can't find the container with id 140e1a7f0a215acba735d91f6a862910ebfd304a819860d0e50f33f515e34c9b
Oct 30 21:13:10.193901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.191443    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"]
Oct 30 21:13:10.233582 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.233554    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw" podUID="fb43ea3e-4370-4b40-b946-32938c66f554" containerName="payload-processing" containerID="cri-o://24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a" gracePeriod=25
Oct 30 21:13:10.238242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.238138    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw" podUID="a059db35-6bd6-47aa-a491-efe1f48606e1" containerName="payload-processing" containerID="cri-o://cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1" gracePeriod=25
Oct 30 21:13:10.521911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.521883    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:13:10.547182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.547156    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:13:10.622658 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.622616    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-provider-credentials\") pod \"fb43ea3e-4370-4b40-b946-32938c66f554\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") "
Oct 30 21:13:10.622837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.622759    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-routing-overlay-volume\") pod \"fb43ea3e-4370-4b40-b946-32938c66f554\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") "
Oct 30 21:13:10.622837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.622802    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-kube-api-access-r659j\" (UniqueName: \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-kube-api-access-r659j\") pod \"fb43ea3e-4370-4b40-b946-32938c66f554\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") "
Oct 30 21:13:10.624408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.623407    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-routing-overlay-volume" pod "fb43ea3e-4370-4b40-b946-32938c66f554" (UID: "fb43ea3e-4370-4b40-b946-32938c66f554"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:10.630938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.626132    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-plugins-config-volume\") pod \"fb43ea3e-4370-4b40-b946-32938c66f554\" (UID: \"fb43ea3e-4370-4b40-b946-32938c66f554\") "
Oct 30 21:13:10.630938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.626270    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-provider-credentials" pod "fb43ea3e-4370-4b40-b946-32938c66f554" (UID: "fb43ea3e-4370-4b40-b946-32938c66f554"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:10.630938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.626610    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-plugins-config-volume" pod "fb43ea3e-4370-4b40-b946-32938c66f554" (UID: "fb43ea3e-4370-4b40-b946-32938c66f554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:10.630938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.628354    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.630938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.628377    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.630938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.628401    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb43ea3e-4370-4b40-b946-32938c66f554-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.631343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.631096    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-kube-api-access-r659j" pod "fb43ea3e-4370-4b40-b946-32938c66f554" (UID: "fb43ea3e-4370-4b40-b946-32938c66f554"). InnerVolumeSpecName "kube-api-access-r659j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:10.642596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.642562    4810 generic.go:356] "Generic (PLEG): container finished" podID="fb43ea3e-4370-4b40-b946-32938c66f554" containerID="24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a" exitCode=0
Oct 30 21:13:10.642722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.642657    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw" event={"ID":"fb43ea3e-4370-4b40-b946-32938c66f554","Type":"ContainerDied","Data":"24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a"}
Oct 30 21:13:10.642793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.642739    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw" event={"ID":"fb43ea3e-4370-4b40-b946-32938c66f554","Type":"ContainerDied","Data":"6e30629a0ce520681f7b74566a8c5f3f398dac402b7e9a3cfe792279cfbfcd8c"}
Oct 30 21:13:10.642793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.642769    4810 scope.go:122] "RemoveContainer" containerID="24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a"
Oct 30 21:13:10.642992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.642957    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"
Oct 30 21:13:10.646191 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.646173    4810 generic.go:356] "Generic (PLEG): container finished" podID="a059db35-6bd6-47aa-a491-efe1f48606e1" containerID="cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1" exitCode=0
Oct 30 21:13:10.646327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.646307    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"
Oct 30 21:13:10.646419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.646260    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw" event={"ID":"a059db35-6bd6-47aa-a491-efe1f48606e1","Type":"ContainerDied","Data":"cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1"}
Oct 30 21:13:10.646498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.646456    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw" event={"ID":"a059db35-6bd6-47aa-a491-efe1f48606e1","Type":"ContainerDied","Data":"fc07b29ea8963441fe4852626ef6b64cebda454ae4b4ee81a3181103e5f07e60"}
Oct 30 21:13:10.648372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.648351    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq" event={"ID":"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb","Type":"ContainerStarted","Data":"e42ecfd52d2e7348948cec99a167b29bafa5960ad84a37d96abd345eb2108c38"}
Oct 30 21:13:10.648488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.648386    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq" event={"ID":"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb","Type":"ContainerStarted","Data":"140e1a7f0a215acba735d91f6a862910ebfd304a819860d0e50f33f515e34c9b"}
Oct 30 21:13:10.648488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.648482    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:10.662018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.661946    4810 scope.go:122] "RemoveContainer" containerID="24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a"
Oct 30 21:13:10.662479 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:13:10.662446    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a\": container with ID starting with 24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a not found: ID does not exist" containerID="24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a"
Oct 30 21:13:10.662561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.662485    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a"} err="failed to get container status \"24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a\": rpc error: code = NotFound desc = could not find container \"24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a\": container with ID starting with 24fef350c518bf2e78ce1ff9e8541a070ea1cca231667001bdd364ad5a44178a not found: ID does not exist"
Oct 30 21:13:10.662561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.662501    4810 scope.go:122] "RemoveContainer" containerID="cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1"
Oct 30 21:13:10.674667 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.674643    4810 scope.go:122] "RemoveContainer" containerID="cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1"
Oct 30 21:13:10.675030 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:13:10.675004    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1\": container with ID starting with cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1 not found: ID does not exist" containerID="cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1"
Oct 30 21:13:10.675103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.675036    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1"} err="failed to get container status \"cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1\": rpc error: code = NotFound desc = could not find container \"cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1\": container with ID starting with cf6caf35dc2360dcd430b53b7b20d74726acef6d5cd0ef50cd3c9b15f6f8d2d1 not found: ID does not exist"
Oct 30 21:13:10.685991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.685951    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq" podStartSLOduration=1.6859379589999999 podStartE2EDuration="1.685937959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:10.680968225 +0000 UTC m=+2157.172784771" watchObservedRunningTime="2026-10-30 21:13:10.685937959 +0000 UTC m=+2157.177754504"
Oct 30 21:13:10.701620 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.701583    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"]
Oct 30 21:13:10.705487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.705463    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-574css4lw"]
Oct 30 21:13:10.742542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.736331    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-kube-api-access-f88z8\" (UniqueName: \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-kube-api-access-f88z8\") pod \"a059db35-6bd6-47aa-a491-efe1f48606e1\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") "
Oct 30 21:13:10.742542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.736475    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-provider-credentials\") pod \"a059db35-6bd6-47aa-a491-efe1f48606e1\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") "
Oct 30 21:13:10.742542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.736531    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-plugins-config-volume\") pod \"a059db35-6bd6-47aa-a491-efe1f48606e1\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") "
Oct 30 21:13:10.742542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.736699    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-routing-overlay-volume\") pod \"a059db35-6bd6-47aa-a491-efe1f48606e1\" (UID: \"a059db35-6bd6-47aa-a491-efe1f48606e1\") "
Oct 30 21:13:10.742542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.737314    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r659j\" (UniqueName: \"kubernetes.io/projected/fb43ea3e-4370-4b40-b946-32938c66f554-kube-api-access-r659j\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.742542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.738602    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-routing-overlay-volume" pod "a059db35-6bd6-47aa-a491-efe1f48606e1" (UID: "a059db35-6bd6-47aa-a491-efe1f48606e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:10.742542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.739258    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-plugins-config-volume" pod "a059db35-6bd6-47aa-a491-efe1f48606e1" (UID: "a059db35-6bd6-47aa-a491-efe1f48606e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:10.749346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.744086    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-provider-credentials" pod "a059db35-6bd6-47aa-a491-efe1f48606e1" (UID: "a059db35-6bd6-47aa-a491-efe1f48606e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:10.749346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.745721    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-kube-api-access-f88z8" pod "a059db35-6bd6-47aa-a491-efe1f48606e1" (UID: "a059db35-6bd6-47aa-a491-efe1f48606e1"). InnerVolumeSpecName "kube-api-access-f88z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:10.839295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.839249    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.839295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.839280    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.839295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.839291    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a059db35-6bd6-47aa-a491-efe1f48606e1-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.839295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.839304    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f88z8\" (UniqueName: \"kubernetes.io/projected/a059db35-6bd6-47aa-a491-efe1f48606e1-kube-api-access-f88z8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.980173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.980140    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"]
Oct 30 21:13:10.988332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:10.988294    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-758bvcwcw"]
Oct 30 21:13:11.665038 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:11.665002    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a059db35-6bd6-47aa-a491-efe1f48606e1" path="/var/lib/kubelet/pods/a059db35-6bd6-47aa-a491-efe1f48606e1/volumes"
Oct 30 21:13:11.665691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:11.665670    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb43ea3e-4370-4b40-b946-32938c66f554" path="/var/lib/kubelet/pods/fb43ea3e-4370-4b40-b946-32938c66f554/volumes"
Oct 30 21:13:12.773722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.773687    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"]
Oct 30 21:13:12.774487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.774472    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a059db35-6bd6-47aa-a491-efe1f48606e1" containerName="payload-processing"
Oct 30 21:13:12.774534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.774489    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a059db35-6bd6-47aa-a491-efe1f48606e1" containerName="payload-processing"
Oct 30 21:13:12.774534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.774505    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fb43ea3e-4370-4b40-b946-32938c66f554" containerName="payload-processing"
Oct 30 21:13:12.774534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.774510    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb43ea3e-4370-4b40-b946-32938c66f554" containerName="payload-processing"
Oct 30 21:13:12.774663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.774653    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a059db35-6bd6-47aa-a491-efe1f48606e1" containerName="payload-processing"
Oct 30 21:13:12.774705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.774666    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb43ea3e-4370-4b40-b946-32938c66f554" containerName="payload-processing"
Oct 30 21:13:12.780460 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.780439    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.791248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.791225    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"agc-cred-a2ff349fd807\""
Oct 30 21:13:12.807606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.807579    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"]
Oct 30 21:13:12.874241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.874199    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.874390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.874374    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.874522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.874497    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.874565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.874547    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4gps\" (UniqueName: \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-kube-api-access-r4gps\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.975791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.975746    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.975983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.975811    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4gps\" (UniqueName: \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-kube-api-access-r4gps\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.975983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.975969    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.976147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.976126    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.976703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.976663    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.976827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.976809    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.978640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.978615    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:12.985988 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:12.985966    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4gps\" (UniqueName: \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-kube-api-access-r4gps\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:13.096654 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:13.096560    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:13.387444 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:13:13.387414    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b886ae_5b1c_43fb_94ff_92c7ac1f3683.slice/crio-4d59b5f33902a460f04e49c13d80e897eacade538ade1d76758f6062a0274aa5 WatchSource:0}: Error finding container 4d59b5f33902a460f04e49c13d80e897eacade538ade1d76758f6062a0274aa5: Status 404 returned error can't find the container with id 4d59b5f33902a460f04e49c13d80e897eacade538ade1d76758f6062a0274aa5
Oct 30 21:13:13.387951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:13.387923    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"]
Oct 30 21:13:13.684936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:13.684833    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97" event={"ID":"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683","Type":"ContainerStarted","Data":"ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621"}
Oct 30 21:13:13.684936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:13.684910    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97" event={"ID":"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683","Type":"ContainerStarted","Data":"4d59b5f33902a460f04e49c13d80e897eacade538ade1d76758f6062a0274aa5"}
Oct 30 21:13:13.685181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:13.684961    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:13.708719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:13.708669    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97" podStartSLOduration=1.708657039 podStartE2EDuration="1.708657039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:13.704723237 +0000 UTC m=+2160.196539783" watchObservedRunningTime="2026-10-30 21:13:13.708657039 +0000 UTC m=+2160.200473583"
Oct 30 21:13:14.894567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:14.894530    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"]
Oct 30 21:13:14.895113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:14.894737    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy" containerID="cri-o://64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f" gracePeriod=30
Oct 30 21:13:16.748535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:16.748496    4810 scope.go:122] "RemoveContainer" containerID="cd6d4aa7bc95d87a0a0c0577a14967c3df7342933891ab1c207c9900a9d85dcd"
Oct 30 21:13:16.762945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:16.762913    4810 scope.go:122] "RemoveContainer" containerID="64e462cd454bed25c42a286f07daba1ab93d8860075aa12df16326711aca4b07"
Oct 30 21:13:16.776982 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:16.776954    4810 scope.go:122] "RemoveContainer" containerID="fab35b2d43cdaf460baabf8189963e0f9603a6cd3a55cd0492a1917c5d8adca8"
Oct 30 21:13:19.000473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.000430    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"]
Oct 30 21:13:19.008479 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.008447    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.014012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.013972    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:19.014968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.014932    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:13:19.018003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.017974    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"]
Oct 30 21:13:19.080460 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.080406    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae547c11-h7fls\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.080671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.080521    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae547c11-h7fls\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.080671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.080550    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdsj\" (UniqueName: \"kubernetes.io/projected/056785f6-bd60-4661-8a82-9143a5ef7487-kube-api-access-6vdsj\") pod \"maas-api-revoke-sub-ae547c11-h7fls\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.181745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.181708    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae547c11-h7fls\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.181944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.181800    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae547c11-h7fls\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.181944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.181830    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdsj\" (UniqueName: \"kubernetes.io/projected/056785f6-bd60-4661-8a82-9143a5ef7487-kube-api-access-6vdsj\") pod \"maas-api-revoke-sub-ae547c11-h7fls\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.182508 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.182483    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae547c11-h7fls\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.184296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.184268    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae547c11-h7fls\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.193213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.193191    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdsj\" (UniqueName: \"kubernetes.io/projected/056785f6-bd60-4661-8a82-9143a5ef7487-kube-api-access-6vdsj\") pod \"maas-api-revoke-sub-ae547c11-h7fls\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.335446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.335366    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:19.628087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.628056    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"]
Oct 30 21:13:19.630453 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:13:19.630423    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056785f6_bd60_4661_8a82_9143a5ef7487.slice/crio-10fac29dcf6d2601ec1805a4826a6f2f8b5b0786540a7518f4b8128b6913dc43 WatchSource:0}: Error finding container 10fac29dcf6d2601ec1805a4826a6f2f8b5b0786540a7518f4b8128b6913dc43: Status 404 returned error can't find the container with id 10fac29dcf6d2601ec1805a4826a6f2f8b5b0786540a7518f4b8128b6913dc43
Oct 30 21:13:19.738254 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:19.738217    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls" event={"ID":"056785f6-bd60-4661-8a82-9143a5ef7487","Type":"ContainerStarted","Data":"10fac29dcf6d2601ec1805a4826a6f2f8b5b0786540a7518f4b8128b6913dc43"}
Oct 30 21:13:20.152410 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.152389    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 21:13:20.197928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.197571    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-data\") pod \"88286263-1069-4357-a702-95d0a60e6db7\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") "
Oct 30 21:13:20.197928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.197692    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-kube-api-access-zbnjn\" (UniqueName: \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-kube-api-access-zbnjn\") pod \"88286263-1069-4357-a702-95d0a60e6db7\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") "
Oct 30 21:13:20.197928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.197766    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-certs\") pod \"88286263-1069-4357-a702-95d0a60e6db7\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") "
Oct 30 21:13:20.198438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.198404    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88286263-1069-4357-a702-95d0a60e6db7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88286263-1069-4357-a702-95d0a60e6db7-istio-podinfo\") pod \"88286263-1069-4357-a702-95d0a60e6db7\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") "
Oct 30 21:13:20.198671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.198603    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-istio-token\" (UniqueName: \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-istio-token\") pod \"88286263-1069-4357-a702-95d0a60e6db7\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") "
Oct 30 21:13:20.198671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.198635    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-envoy\") pod \"88286263-1069-4357-a702-95d0a60e6db7\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") "
Oct 30 21:13:20.198952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.198883    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-credential-socket\") pod \"88286263-1069-4357-a702-95d0a60e6db7\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") "
Oct 30 21:13:20.199953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.199038    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-socket\") pod \"88286263-1069-4357-a702-95d0a60e6db7\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") "
Oct 30 21:13:20.199953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.199071    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88286263-1069-4357-a702-95d0a60e6db7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88286263-1069-4357-a702-95d0a60e6db7-istiod-ca-cert\") pod \"88286263-1069-4357-a702-95d0a60e6db7\" (UID: \"88286263-1069-4357-a702-95d0a60e6db7\") "
Oct 30 21:13:20.201156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.200908    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88286263-1069-4357-a702-95d0a60e6db7-istiod-ca-cert" pod "88286263-1069-4357-a702-95d0a60e6db7" (UID: "88286263-1069-4357-a702-95d0a60e6db7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:20.201554 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.201282    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-data" pod "88286263-1069-4357-a702-95d0a60e6db7" (UID: "88286263-1069-4357-a702-95d0a60e6db7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:13:20.202553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.202529    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-socket" pod "88286263-1069-4357-a702-95d0a60e6db7" (UID: "88286263-1069-4357-a702-95d0a60e6db7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:13:20.202818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.202794    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-credential-socket" pod "88286263-1069-4357-a702-95d0a60e6db7" (UID: "88286263-1069-4357-a702-95d0a60e6db7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:13:20.203160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.203133    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-certs" pod "88286263-1069-4357-a702-95d0a60e6db7" (UID: "88286263-1069-4357-a702-95d0a60e6db7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:13:20.205583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.205256    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-envoy" pod "88286263-1069-4357-a702-95d0a60e6db7" (UID: "88286263-1069-4357-a702-95d0a60e6db7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:13:20.211324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.211250    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-istio-token" pod "88286263-1069-4357-a702-95d0a60e6db7" (UID: "88286263-1069-4357-a702-95d0a60e6db7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:20.211324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.211286    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-kube-api-access-zbnjn" pod "88286263-1069-4357-a702-95d0a60e6db7" (UID: "88286263-1069-4357-a702-95d0a60e6db7"). InnerVolumeSpecName "kube-api-access-zbnjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:20.213410 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.213384    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88286263-1069-4357-a702-95d0a60e6db7-istio-podinfo" pod "88286263-1069-4357-a702-95d0a60e6db7" (UID: "88286263-1069-4357-a702-95d0a60e6db7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:13:20.301271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.301237    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:20.301271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.301270    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbnjn\" (UniqueName: \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-kube-api-access-zbnjn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:20.301529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.301282    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:20.301529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.301291    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88286263-1069-4357-a702-95d0a60e6db7-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:20.301529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.301300    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88286263-1069-4357-a702-95d0a60e6db7-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:20.301529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.301308    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:20.301529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.301317    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:20.301529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.301331    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88286263-1069-4357-a702-95d0a60e6db7-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:20.301529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.301341    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88286263-1069-4357-a702-95d0a60e6db7-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:20.747486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.747447    4810 generic.go:356] "Generic (PLEG): container finished" podID="056785f6-bd60-4661-8a82-9143a5ef7487" containerID="454dbcadc051761f69f9bade60dd50052abfcd8c6cdaeaa641ecc7ca9feb2ce6" exitCode=0
Oct 30 21:13:20.747654 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.747528    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls" event={"ID":"056785f6-bd60-4661-8a82-9143a5ef7487","Type":"ContainerDied","Data":"454dbcadc051761f69f9bade60dd50052abfcd8c6cdaeaa641ecc7ca9feb2ce6"}
Oct 30 21:13:20.749118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.749094    4810 generic.go:356] "Generic (PLEG): container finished" podID="88286263-1069-4357-a702-95d0a60e6db7" containerID="64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f" exitCode=0
Oct 30 21:13:20.749234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.749142    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"
Oct 30 21:13:20.749234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.749175    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" event={"ID":"88286263-1069-4357-a702-95d0a60e6db7","Type":"ContainerDied","Data":"64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f"}
Oct 30 21:13:20.749234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.749219    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k" event={"ID":"88286263-1069-4357-a702-95d0a60e6db7","Type":"ContainerDied","Data":"b0a5d5830d0755759a33428ffa5f677b4314b9113a4510bd74fb677a1f242c6d"}
Oct 30 21:13:20.749403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.749242    4810 scope.go:122] "RemoveContainer" containerID="64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f"
Oct 30 21:13:20.761217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.761194    4810 scope.go:122] "RemoveContainer" containerID="64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f"
Oct 30 21:13:20.761505 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:13:20.761483    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f\": container with ID starting with 64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f not found: ID does not exist" containerID="64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f"
Oct 30 21:13:20.761564 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.761513    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f"} err="failed to get container status \"64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f\": rpc error: code = NotFound desc = could not find container \"64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f\": container with ID starting with 64ee7729e7d120f2c727610b855f0fb108a4c2b31b32ca84b855d10cca26b62f not found: ID does not exist"
Oct 30 21:13:20.793736 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.793327    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"]
Oct 30 21:13:20.797690 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:20.797658    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-x6w6k"]
Oct 30 21:13:21.665246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:21.665209    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88286263-1069-4357-a702-95d0a60e6db7" path="/var/lib/kubelet/pods/88286263-1069-4357-a702-95d0a60e6db7/volumes"
Oct 30 21:13:21.665739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:21.665723    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:13:21.797308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:21.797262    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"]
Oct 30 21:13:21.955243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:21.955217    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:22.029193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.029152    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056785f6-bd60-4661-8a82-9143a5ef7487-kube-api-access-6vdsj\" (UniqueName: \"kubernetes.io/projected/056785f6-bd60-4661-8a82-9143a5ef7487-kube-api-access-6vdsj\") pod \"056785f6-bd60-4661-8a82-9143a5ef7487\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") "
Oct 30 21:13:22.029384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.029241    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-xks-ca\") pod \"056785f6-bd60-4661-8a82-9143a5ef7487\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") "
Oct 30 21:13:22.029384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.029306    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-openshift-ca\") pod \"056785f6-bd60-4661-8a82-9143a5ef7487\" (UID: \"056785f6-bd60-4661-8a82-9143a5ef7487\") "
Oct 30 21:13:22.031405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.030686    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-openshift-ca" pod "056785f6-bd60-4661-8a82-9143a5ef7487" (UID: "056785f6-bd60-4661-8a82-9143a5ef7487"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:22.035193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.034668    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056785f6-bd60-4661-8a82-9143a5ef7487-kube-api-access-6vdsj" pod "056785f6-bd60-4661-8a82-9143a5ef7487" (UID: "056785f6-bd60-4661-8a82-9143a5ef7487"). InnerVolumeSpecName "kube-api-access-6vdsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:22.035467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.035433    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-xks-ca" pod "056785f6-bd60-4661-8a82-9143a5ef7487" (UID: "056785f6-bd60-4661-8a82-9143a5ef7487"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:22.131873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.131816    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdsj\" (UniqueName: \"kubernetes.io/projected/056785f6-bd60-4661-8a82-9143a5ef7487-kube-api-access-6vdsj\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:22.132052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.131894    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:22.132052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.131911    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056785f6-bd60-4661-8a82-9143a5ef7487-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:22.769607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.769575    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"
Oct 30 21:13:22.770106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.769603    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls" event={"ID":"056785f6-bd60-4661-8a82-9143a5ef7487","Type":"ContainerDied","Data":"10fac29dcf6d2601ec1805a4826a6f2f8b5b0786540a7518f4b8128b6913dc43"}
Oct 30 21:13:22.770106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:22.769651    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10fac29dcf6d2601ec1805a4826a6f2f8b5b0786540a7518f4b8128b6913dc43"
Oct 30 21:13:24.318063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.318021    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"]
Oct 30 21:13:24.319275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.319247    4810 cpu_manager.go:414] "containerMap: removing container" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy"
Oct 30 21:13:24.319275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.319273    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy"
Oct 30 21:13:24.319477 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.319313    4810 cpu_manager.go:414] "containerMap: removing container" podUID="056785f6-bd60-4661-8a82-9143a5ef7487" containerName="revoke-keys"
Oct 30 21:13:24.319477 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.319322    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056785f6-bd60-4661-8a82-9143a5ef7487" containerName="revoke-keys"
Oct 30 21:13:24.319545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.319488    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="056785f6-bd60-4661-8a82-9143a5ef7487" containerName="revoke-keys"
Oct 30 21:13:24.319545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.319515    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="88286263-1069-4357-a702-95d0a60e6db7" containerName="istio-proxy"
Oct 30 21:13:24.322967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.322943    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.330074 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.330049    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"kube-root-ca.crt\""
Oct 30 21:13:24.330262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.330062    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"routing-overlay\""
Oct 30 21:13:24.330383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.330112    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"openshift-service-ca.crt\""
Oct 30 21:13:24.330452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.330172    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"payload-processing-external-model-e2e--a89b3f58-dockercfg-cssz9\""
Oct 30 21:13:24.330452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.330201    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"agc-cred-cd28d0b544a0\""
Oct 30 21:13:24.330452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.330270    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6741b1\""
Oct 30 21:13:24.332086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.332065    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"]
Oct 30 21:13:24.495933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.494831    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.495933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.494930    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.495933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.495043    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.495933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.495069    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ksft\" (UniqueName: \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-kube-api-access-6ksft\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.597938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.597083    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.597938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.597167    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.597938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.597281    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.597938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.597308    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksft\" (UniqueName: \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-kube-api-access-6ksft\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.598885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.598739    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.598885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.598807    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.601549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.601522    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.618721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.618685    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksft\" (UniqueName: \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-kube-api-access-6ksft\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.648399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.648113    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:24.654022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.653951    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"]
Oct 30 21:13:24.660101 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.659289    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.671702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.666126    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"agc-cred-8bbfc79e2e3a\""
Oct 30 21:13:24.673103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.673068    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"]
Oct 30 21:13:24.699253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.699225    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:13:24.757532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.757498    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"]
Oct 30 21:13:24.812087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.812049    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.812449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.812417    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.812596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.812577    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.812670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.812649    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnhr\" (UniqueName: \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-kube-api-access-zdnhr\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.914540 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.914502    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.914702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.914556    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.914702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.914641    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.914702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.914696    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnhr\" (UniqueName: \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-kube-api-access-zdnhr\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.915280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.915256    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.915429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.915408    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.917193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.917171    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.941901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.941852    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnhr\" (UniqueName: \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-kube-api-access-zdnhr\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:24.982994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:24.982957    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"]
Oct 30 21:13:24.985481 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:13:24.985449    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0e546b_643f_4f91_8060_30b6d1947681.slice/crio-df51678f56f7e9b17890b1c9272f46dd46ede31bfe393f93b8a25d29bb26b83f WatchSource:0}: Error finding container df51678f56f7e9b17890b1c9272f46dd46ede31bfe393f93b8a25d29bb26b83f: Status 404 returned error can't find the container with id df51678f56f7e9b17890b1c9272f46dd46ede31bfe393f93b8a25d29bb26b83f
Oct 30 21:13:25.018806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:25.018786    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:25.166847 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:13:25.166760    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:13:25.314428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:25.314392    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"]
Oct 30 21:13:25.316114 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:13:25.316072    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6efebf_75e1_40c3_a8ff_cc6d033c2100.slice/crio-a7501e8a8202e613180a8d35d9939cbc07243e1c8126c10f43b768d1b89fd05e WatchSource:0}: Error finding container a7501e8a8202e613180a8d35d9939cbc07243e1c8126c10f43b768d1b89fd05e: Status 404 returned error can't find the container with id a7501e8a8202e613180a8d35d9939cbc07243e1c8126c10f43b768d1b89fd05e
Oct 30 21:13:25.842808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:25.842767    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2" event={"ID":"ed6efebf-75e1-40c3-a8ff-cc6d033c2100","Type":"ContainerStarted","Data":"95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13"}
Oct 30 21:13:25.843302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:25.842834    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2" event={"ID":"ed6efebf-75e1-40c3-a8ff-cc6d033c2100","Type":"ContainerStarted","Data":"a7501e8a8202e613180a8d35d9939cbc07243e1c8126c10f43b768d1b89fd05e"}
Oct 30 21:13:25.843302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:25.843065    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:25.844913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:25.844885    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b" event={"ID":"2d0e546b-643f-4f91-8060-30b6d1947681","Type":"ContainerStarted","Data":"549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602"}
Oct 30 21:13:25.845052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:25.844931    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b" event={"ID":"2d0e546b-643f-4f91-8060-30b6d1947681","Type":"ContainerStarted","Data":"df51678f56f7e9b17890b1c9272f46dd46ede31bfe393f93b8a25d29bb26b83f"}
Oct 30 21:13:25.845052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:25.845005    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:25.862495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:25.862440    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2" podStartSLOduration=1.8624278429999999 podStartE2EDuration="1.862427843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:25.860459759 +0000 UTC m=+2172.352276306" watchObservedRunningTime="2026-10-30 21:13:25.862427843 +0000 UTC m=+2172.354244416"
Oct 30 21:13:25.879057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:25.879003    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b" podStartSLOduration=1.878988118 podStartE2EDuration="1.878988118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:25.877090737 +0000 UTC m=+2172.368907281" watchObservedRunningTime="2026-10-30 21:13:25.878988118 +0000 UTC m=+2172.370804657"
Oct 30 21:13:26.798199 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:26.798116    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956" podUID="6c940b8d-16eb-4c27-89f5-cf93b7d9958d" containerName="payload-processing" containerID="cri-o://2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5" gracePeriod=25
Oct 30 21:13:27.102709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.102682    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:13:27.251267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.251237    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-routing-overlay-volume\") pod \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") "
Oct 30 21:13:27.251427 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.251354    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-provider-credentials\") pod \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") "
Oct 30 21:13:27.251521 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.251482    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-kube-api-access-gqdsv\" (UniqueName: \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-kube-api-access-gqdsv\") pod \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") "
Oct 30 21:13:27.251589 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.251571    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-plugins-config-volume\") pod \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\" (UID: \"6c940b8d-16eb-4c27-89f5-cf93b7d9958d\") "
Oct 30 21:13:27.251698 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.251672    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-routing-overlay-volume" pod "6c940b8d-16eb-4c27-89f5-cf93b7d9958d" (UID: "6c940b8d-16eb-4c27-89f5-cf93b7d9958d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:27.252085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.252060    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-plugins-config-volume" pod "6c940b8d-16eb-4c27-89f5-cf93b7d9958d" (UID: "6c940b8d-16eb-4c27-89f5-cf93b7d9958d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:27.252444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.252424    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:27.252444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.252445    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:27.253713 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.253689    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-provider-credentials" pod "6c940b8d-16eb-4c27-89f5-cf93b7d9958d" (UID: "6c940b8d-16eb-4c27-89f5-cf93b7d9958d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:27.253811 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.253726    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-kube-api-access-gqdsv" pod "6c940b8d-16eb-4c27-89f5-cf93b7d9958d" (UID: "6c940b8d-16eb-4c27-89f5-cf93b7d9958d"). InnerVolumeSpecName "kube-api-access-gqdsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:27.354234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.354164    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:27.354234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.354191    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqdsv\" (UniqueName: \"kubernetes.io/projected/6c940b8d-16eb-4c27-89f5-cf93b7d9958d-kube-api-access-gqdsv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:27.861609 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.861573    4810 generic.go:356] "Generic (PLEG): container finished" podID="6c940b8d-16eb-4c27-89f5-cf93b7d9958d" containerID="2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5" exitCode=0
Oct 30 21:13:27.861774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.861622    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"
Oct 30 21:13:27.861774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.861655    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956" event={"ID":"6c940b8d-16eb-4c27-89f5-cf93b7d9958d","Type":"ContainerDied","Data":"2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5"}
Oct 30 21:13:27.861774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.861713    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956" event={"ID":"6c940b8d-16eb-4c27-89f5-cf93b7d9958d","Type":"ContainerDied","Data":"57c6b2bdb577857a0cca6238ea6ac572f69985d5ee0c7168abf5f302f6de12a2"}
Oct 30 21:13:27.861774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.861735    4810 scope.go:122] "RemoveContainer" containerID="2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5"
Oct 30 21:13:27.872767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.872745    4810 scope.go:122] "RemoveContainer" containerID="2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5"
Oct 30 21:13:27.873064 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:13:27.873045    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5\": container with ID starting with 2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5 not found: ID does not exist" containerID="2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5"
Oct 30 21:13:27.873163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.873070    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5"} err="failed to get container status \"2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5\": rpc error: code = NotFound desc = could not find container \"2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5\": container with ID starting with 2eebc029e34ca25201d5413908d6bab2ba02ba18b147bfe5d8c5d12114193de5 not found: ID does not exist"
Oct 30 21:13:27.887509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.887482    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"]
Oct 30 21:13:27.892295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:27.892274    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-7f95nh956"]
Oct 30 21:13:29.673600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:29.673561    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c940b8d-16eb-4c27-89f5-cf93b7d9958d" path="/var/lib/kubelet/pods/6c940b8d-16eb-4c27-89f5-cf93b7d9958d/volumes"
Oct 30 21:13:29.758436 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:29.758377    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs" podUID="33df6252-a981-4979-b5e3-0cbdf4936a5a" containerName="payload-processing" containerID="cri-o://fc176f93dfd0597000bf2611730d8a9a1882fffddec7d30cc2f785c6f0ea5507" gracePeriod=25
Oct 30 21:13:29.884120 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:29.884086    4810 generic.go:356] "Generic (PLEG): container finished" podID="33df6252-a981-4979-b5e3-0cbdf4936a5a" containerID="fc176f93dfd0597000bf2611730d8a9a1882fffddec7d30cc2f785c6f0ea5507" exitCode=0
Oct 30 21:13:29.884270 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:29.884126    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs" event={"ID":"33df6252-a981-4979-b5e3-0cbdf4936a5a","Type":"ContainerDied","Data":"fc176f93dfd0597000bf2611730d8a9a1882fffddec7d30cc2f785c6f0ea5507"}
Oct 30 21:13:30.115045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.115012    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:13:30.190756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.190673    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-routing-overlay-volume\") pod \"33df6252-a981-4979-b5e3-0cbdf4936a5a\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") "
Oct 30 21:13:30.191080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.191060    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-plugins-config-volume\") pod \"33df6252-a981-4979-b5e3-0cbdf4936a5a\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") "
Oct 30 21:13:30.191293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.191278    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-provider-credentials\") pod \"33df6252-a981-4979-b5e3-0cbdf4936a5a\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") "
Oct 30 21:13:30.191824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.191804    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-kube-api-access-h5nt2\" (UniqueName: \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-kube-api-access-h5nt2\") pod \"33df6252-a981-4979-b5e3-0cbdf4936a5a\" (UID: \"33df6252-a981-4979-b5e3-0cbdf4936a5a\") "
Oct 30 21:13:30.193008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.191090    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-routing-overlay-volume" pod "33df6252-a981-4979-b5e3-0cbdf4936a5a" (UID: "33df6252-a981-4979-b5e3-0cbdf4936a5a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:30.193191 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.191502    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-plugins-config-volume" pod "33df6252-a981-4979-b5e3-0cbdf4936a5a" (UID: "33df6252-a981-4979-b5e3-0cbdf4936a5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:30.194218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.194172    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-provider-credentials" pod "33df6252-a981-4979-b5e3-0cbdf4936a5a" (UID: "33df6252-a981-4979-b5e3-0cbdf4936a5a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:30.194503 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.194476    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-kube-api-access-h5nt2" pod "33df6252-a981-4979-b5e3-0cbdf4936a5a" (UID: "33df6252-a981-4979-b5e3-0cbdf4936a5a"). InnerVolumeSpecName "kube-api-access-h5nt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:30.294305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.294273    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:30.294305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.294301    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33df6252-a981-4979-b5e3-0cbdf4936a5a-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:30.294565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.294314    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:30.294565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.294328    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5nt2\" (UniqueName: \"kubernetes.io/projected/33df6252-a981-4979-b5e3-0cbdf4936a5a-kube-api-access-h5nt2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:30.362988 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.362960    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"]
Oct 30 21:13:30.363736 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.363723    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6c940b8d-16eb-4c27-89f5-cf93b7d9958d" containerName="payload-processing"
Oct 30 21:13:30.363794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.363738    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c940b8d-16eb-4c27-89f5-cf93b7d9958d" containerName="payload-processing"
Oct 30 21:13:30.363794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.363767    4810 cpu_manager.go:414] "containerMap: removing container" podUID="33df6252-a981-4979-b5e3-0cbdf4936a5a" containerName="payload-processing"
Oct 30 21:13:30.363794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.363772    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33df6252-a981-4979-b5e3-0cbdf4936a5a" containerName="payload-processing"
Oct 30 21:13:30.363930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.363920    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c940b8d-16eb-4c27-89f5-cf93b7d9958d" containerName="payload-processing"
Oct 30 21:13:30.363970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.363936    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="33df6252-a981-4979-b5e3-0cbdf4936a5a" containerName="payload-processing"
Oct 30 21:13:30.366656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.366637    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.371164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.371138    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:30.371604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.371585    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:13:30.377347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.377320    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"]
Oct 30 21:13:30.497323 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.497226    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfphg\" (UniqueName: \"kubernetes.io/projected/9292ee4b-2272-41ad-afcb-3757e6f93df4-kube-api-access-bfphg\") pod \"maas-api-revoke-sub-17241f60-cstqt\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.497500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.497401    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17241f60-cstqt\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.497561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.497537    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17241f60-cstqt\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.599115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.599075    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17241f60-cstqt\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.599696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.599671    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfphg\" (UniqueName: \"kubernetes.io/projected/9292ee4b-2272-41ad-afcb-3757e6f93df4-kube-api-access-bfphg\") pod \"maas-api-revoke-sub-17241f60-cstqt\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.599837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.599768    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17241f60-cstqt\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.600610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.600577    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17241f60-cstqt\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.602600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.602576    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17241f60-cstqt\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.609489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.609464    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfphg\" (UniqueName: \"kubernetes.io/projected/9292ee4b-2272-41ad-afcb-3757e6f93df4-kube-api-access-bfphg\") pod \"maas-api-revoke-sub-17241f60-cstqt\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.685709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.685672    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:30.951055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.951020    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"
Oct 30 21:13:30.951258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.951175    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs" event={"ID":"33df6252-a981-4979-b5e3-0cbdf4936a5a","Type":"ContainerDied","Data":"91450b9782ce6314803e44404bcf03cd4cf97846f721dcbacd3d1eb833fd6954"}
Oct 30 21:13:30.951258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.951242    4810 scope.go:122] "RemoveContainer" containerID="fc176f93dfd0597000bf2611730d8a9a1882fffddec7d30cc2f785c6f0ea5507"
Oct 30 21:13:30.982956 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.982917    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"]
Oct 30 21:13:30.987918 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:13:30.985076    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9292ee4b_2272_41ad_afcb_3757e6f93df4.slice/crio-503f12baa82e01975560dd05421a7faf917c5f4658d52fa730c095941038fdf3 WatchSource:0}: Error finding container 503f12baa82e01975560dd05421a7faf917c5f4658d52fa730c095941038fdf3: Status 404 returned error can't find the container with id 503f12baa82e01975560dd05421a7faf917c5f4658d52fa730c095941038fdf3
Oct 30 21:13:30.990304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.990283    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"]
Oct 30 21:13:30.993437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:30.993413    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dv5vbs"]
Oct 30 21:13:31.664312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:31.664285    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33df6252-a981-4979-b5e3-0cbdf4936a5a" path="/var/lib/kubelet/pods/33df6252-a981-4979-b5e3-0cbdf4936a5a/volumes"
Oct 30 21:13:31.961787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:31.961714    4810 generic.go:356] "Generic (PLEG): container finished" podID="9292ee4b-2272-41ad-afcb-3757e6f93df4" containerID="19a06a6e1d7bfa49d701d07a3af5c350c583c3ceea270aaf3b215184a518c5e8" exitCode=0
Oct 30 21:13:31.962206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:31.961800    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt" event={"ID":"9292ee4b-2272-41ad-afcb-3757e6f93df4","Type":"ContainerDied","Data":"19a06a6e1d7bfa49d701d07a3af5c350c583c3ceea270aaf3b215184a518c5e8"}
Oct 30 21:13:31.962206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:31.961851    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt" event={"ID":"9292ee4b-2272-41ad-afcb-3757e6f93df4","Type":"ContainerStarted","Data":"503f12baa82e01975560dd05421a7faf917c5f4658d52fa730c095941038fdf3"}
Oct 30 21:13:33.148621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.148595    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:33.242476 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.242382    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-xks-ca\") pod \"9292ee4b-2272-41ad-afcb-3757e6f93df4\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") "
Oct 30 21:13:33.242476 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.242455    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9292ee4b-2272-41ad-afcb-3757e6f93df4-kube-api-access-bfphg\" (UniqueName: \"kubernetes.io/projected/9292ee4b-2272-41ad-afcb-3757e6f93df4-kube-api-access-bfphg\") pod \"9292ee4b-2272-41ad-afcb-3757e6f93df4\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") "
Oct 30 21:13:33.242722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.242559    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-openshift-ca\") pod \"9292ee4b-2272-41ad-afcb-3757e6f93df4\" (UID: \"9292ee4b-2272-41ad-afcb-3757e6f93df4\") "
Oct 30 21:13:33.242968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.242944    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-openshift-ca" pod "9292ee4b-2272-41ad-afcb-3757e6f93df4" (UID: "9292ee4b-2272-41ad-afcb-3757e6f93df4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:33.244514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.244487    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-xks-ca" pod "9292ee4b-2272-41ad-afcb-3757e6f93df4" (UID: "9292ee4b-2272-41ad-afcb-3757e6f93df4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:33.244630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.244612    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9292ee4b-2272-41ad-afcb-3757e6f93df4-kube-api-access-bfphg" pod "9292ee4b-2272-41ad-afcb-3757e6f93df4" (UID: "9292ee4b-2272-41ad-afcb-3757e6f93df4"). InnerVolumeSpecName "kube-api-access-bfphg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:33.344286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.344229    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:33.344465 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.344289    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfphg\" (UniqueName: \"kubernetes.io/projected/9292ee4b-2272-41ad-afcb-3757e6f93df4-kube-api-access-bfphg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:33.344465 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.344316    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9292ee4b-2272-41ad-afcb-3757e6f93df4-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:33.979495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.979405    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt" event={"ID":"9292ee4b-2272-41ad-afcb-3757e6f93df4","Type":"ContainerDied","Data":"503f12baa82e01975560dd05421a7faf917c5f4658d52fa730c095941038fdf3"}
Oct 30 21:13:33.979495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.979452    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="503f12baa82e01975560dd05421a7faf917c5f4658d52fa730c095941038fdf3"
Oct 30 21:13:33.979495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:33.979420    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"
Oct 30 21:13:36.854898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:36.854851    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:13:36.855285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:36.854943    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:36.883685 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:36.883652    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:13:36.937505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:36.937463    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"]
Oct 30 21:13:40.829484 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.829447    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"]
Oct 30 21:13:40.830283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.830266    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9292ee4b-2272-41ad-afcb-3757e6f93df4" containerName="revoke-keys"
Oct 30 21:13:40.830328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.830284    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9292ee4b-2272-41ad-afcb-3757e6f93df4" containerName="revoke-keys"
Oct 30 21:13:40.830401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.830392    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9292ee4b-2272-41ad-afcb-3757e6f93df4" containerName="revoke-keys"
Oct 30 21:13:40.834250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.834235    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:40.838747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.838716    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:40.839416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.839224    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:13:40.845494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.845346    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"]
Oct 30 21:13:40.964399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.964365    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a633ccfb-4lb5j\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:40.964556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.964407    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7fhl\" (UniqueName: \"kubernetes.io/projected/7158dcac-f804-4aaa-92bb-e6a83f5056b2-kube-api-access-c7fhl\") pod \"maas-api-revoke-sub-a633ccfb-4lb5j\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:40.964556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:40.964480    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a633ccfb-4lb5j\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:41.065456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:41.065426    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a633ccfb-4lb5j\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:41.065595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:41.065466    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7fhl\" (UniqueName: \"kubernetes.io/projected/7158dcac-f804-4aaa-92bb-e6a83f5056b2-kube-api-access-c7fhl\") pod \"maas-api-revoke-sub-a633ccfb-4lb5j\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:41.065728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:41.065709    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a633ccfb-4lb5j\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:41.066093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:41.066068    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a633ccfb-4lb5j\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:41.068063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:41.068040    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a633ccfb-4lb5j\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:41.076164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:41.076139    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7fhl\" (UniqueName: \"kubernetes.io/projected/7158dcac-f804-4aaa-92bb-e6a83f5056b2-kube-api-access-c7fhl\") pod \"maas-api-revoke-sub-a633ccfb-4lb5j\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:41.151242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:41.151220    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:41.436423 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:41.436384    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"]
Oct 30 21:13:41.436567 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:13:41.436544    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7158dcac_f804_4aaa_92bb_e6a83f5056b2.slice/crio-ab474421b1fff6c0fa7dcf108f291887b638c670ec0fa86c71473fe29b4fe906 WatchSource:0}: Error finding container ab474421b1fff6c0fa7dcf108f291887b638c670ec0fa86c71473fe29b4fe906: Status 404 returned error can't find the container with id ab474421b1fff6c0fa7dcf108f291887b638c670ec0fa86c71473fe29b4fe906
Oct 30 21:13:42.007755 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.007664    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b" podUID="2d0e546b-643f-4f91-8060-30b6d1947681" containerName="payload-processing" containerID="cri-o://549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602" gracePeriod=25
Oct 30 21:13:42.055939 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.053794    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j" event={"ID":"7158dcac-f804-4aaa-92bb-e6a83f5056b2","Type":"ContainerStarted","Data":"380089cd5b50a47bf24e933478c403443ffe4a30c072cec7ca1920e3facd5cd8"}
Oct 30 21:13:42.055939 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.053941    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j" event={"ID":"7158dcac-f804-4aaa-92bb-e6a83f5056b2","Type":"ContainerStarted","Data":"ab474421b1fff6c0fa7dcf108f291887b638c670ec0fa86c71473fe29b4fe906"}
Oct 30 21:13:42.076462 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.076408    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j" podStartSLOduration=2.076390504 podStartE2EDuration="2.076390504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:42.071521293 +0000 UTC m=+2188.563337839" watchObservedRunningTime="2026-10-30 21:13:42.076390504 +0000 UTC m=+2188.568207050"
Oct 30 21:13:42.389100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.389073    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:42.487995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.487964    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-plugins-config-volume\") pod \"2d0e546b-643f-4f91-8060-30b6d1947681\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") "
Oct 30 21:13:42.488174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.488039    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-kube-api-access-6ksft\" (UniqueName: \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-kube-api-access-6ksft\") pod \"2d0e546b-643f-4f91-8060-30b6d1947681\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") "
Oct 30 21:13:42.488250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.488233    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-provider-credentials\") pod \"2d0e546b-643f-4f91-8060-30b6d1947681\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") "
Oct 30 21:13:42.488289 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.488272    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-routing-overlay-volume\") pod \"2d0e546b-643f-4f91-8060-30b6d1947681\" (UID: \"2d0e546b-643f-4f91-8060-30b6d1947681\") "
Oct 30 21:13:42.488432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.488399    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-plugins-config-volume" pod "2d0e546b-643f-4f91-8060-30b6d1947681" (UID: "2d0e546b-643f-4f91-8060-30b6d1947681"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:42.488919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.488842    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-routing-overlay-volume" pod "2d0e546b-643f-4f91-8060-30b6d1947681" (UID: "2d0e546b-643f-4f91-8060-30b6d1947681"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:42.488919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.488890    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:42.490510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.490487    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-kube-api-access-6ksft" pod "2d0e546b-643f-4f91-8060-30b6d1947681" (UID: "2d0e546b-643f-4f91-8060-30b6d1947681"). InnerVolumeSpecName "kube-api-access-6ksft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:42.490510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.490520    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-provider-credentials" pod "2d0e546b-643f-4f91-8060-30b6d1947681" (UID: "2d0e546b-643f-4f91-8060-30b6d1947681"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:42.590554 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.590466    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ksft\" (UniqueName: \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-kube-api-access-6ksft\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:42.590554 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.590499    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d0e546b-643f-4f91-8060-30b6d1947681-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:42.590554 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:42.590512    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d0e546b-643f-4f91-8060-30b6d1947681-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:43.067470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.067436    4810 generic.go:356] "Generic (PLEG): container finished" podID="7158dcac-f804-4aaa-92bb-e6a83f5056b2" containerID="380089cd5b50a47bf24e933478c403443ffe4a30c072cec7ca1920e3facd5cd8" exitCode=0
Oct 30 21:13:43.068134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.067514    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j" event={"ID":"7158dcac-f804-4aaa-92bb-e6a83f5056b2","Type":"ContainerDied","Data":"380089cd5b50a47bf24e933478c403443ffe4a30c072cec7ca1920e3facd5cd8"}
Oct 30 21:13:43.075688 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.075658    4810 generic.go:356] "Generic (PLEG): container finished" podID="2d0e546b-643f-4f91-8060-30b6d1947681" containerID="549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602" exitCode=0
Oct 30 21:13:43.075950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.075923    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b" event={"ID":"2d0e546b-643f-4f91-8060-30b6d1947681","Type":"ContainerDied","Data":"549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602"}
Oct 30 21:13:43.076060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.075971    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b" event={"ID":"2d0e546b-643f-4f91-8060-30b6d1947681","Type":"ContainerDied","Data":"df51678f56f7e9b17890b1c9272f46dd46ede31bfe393f93b8a25d29bb26b83f"}
Oct 30 21:13:43.076060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.075980    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"
Oct 30 21:13:43.076060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.076000    4810 scope.go:122] "RemoveContainer" containerID="549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602"
Oct 30 21:13:43.091619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.091599    4810 scope.go:122] "RemoveContainer" containerID="549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602"
Oct 30 21:13:43.098985 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:13:43.098947    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602\": container with ID starting with 549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602 not found: ID does not exist" containerID="549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602"
Oct 30 21:13:43.099122 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.098991    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602"} err="failed to get container status \"549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602\": rpc error: code = NotFound desc = could not find container \"549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602\": container with ID starting with 549fe0c228fbdd8d65a5a1cd951c8bdf7ece6f2aadf55e7f04665bcfc4ed6602 not found: ID does not exist"
Oct 30 21:13:43.122971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.122938    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"]
Oct 30 21:13:43.125846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.125824    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799wrv4b"]
Oct 30 21:13:43.663926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:43.663899    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0e546b-643f-4f91-8060-30b6d1947681" path="/var/lib/kubelet/pods/2d0e546b-643f-4f91-8060-30b6d1947681/volumes"
Oct 30 21:13:44.260109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:44.260082    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:44.417371 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:44.417339    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-openshift-ca\") pod \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") "
Oct 30 21:13:44.417592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:44.417460    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7158dcac-f804-4aaa-92bb-e6a83f5056b2-kube-api-access-c7fhl\" (UniqueName: \"kubernetes.io/projected/7158dcac-f804-4aaa-92bb-e6a83f5056b2-kube-api-access-c7fhl\") pod \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") "
Oct 30 21:13:44.417592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:44.417574    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-xks-ca\") pod \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\" (UID: \"7158dcac-f804-4aaa-92bb-e6a83f5056b2\") "
Oct 30 21:13:44.417730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:44.417703    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-openshift-ca" pod "7158dcac-f804-4aaa-92bb-e6a83f5056b2" (UID: "7158dcac-f804-4aaa-92bb-e6a83f5056b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:44.418142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:44.418119    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:44.419742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:44.419714    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7158dcac-f804-4aaa-92bb-e6a83f5056b2-kube-api-access-c7fhl" pod "7158dcac-f804-4aaa-92bb-e6a83f5056b2" (UID: "7158dcac-f804-4aaa-92bb-e6a83f5056b2"). InnerVolumeSpecName "kube-api-access-c7fhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:44.421958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:44.421937    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-xks-ca" pod "7158dcac-f804-4aaa-92bb-e6a83f5056b2" (UID: "7158dcac-f804-4aaa-92bb-e6a83f5056b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:44.519399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:44.519373    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158dcac-f804-4aaa-92bb-e6a83f5056b2-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:44.519539 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:44.519403    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7fhl\" (UniqueName: \"kubernetes.io/projected/7158dcac-f804-4aaa-92bb-e6a83f5056b2-kube-api-access-c7fhl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:45.095722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:45.095687    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j" event={"ID":"7158dcac-f804-4aaa-92bb-e6a83f5056b2","Type":"ContainerDied","Data":"ab474421b1fff6c0fa7dcf108f291887b638c670ec0fa86c71473fe29b4fe906"}
Oct 30 21:13:45.095919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:45.095741    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab474421b1fff6c0fa7dcf108f291887b638c670ec0fa86c71473fe29b4fe906"
Oct 30 21:13:45.095919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:45.095738    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"
Oct 30 21:13:54.120693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.120651    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"]
Oct 30 21:13:54.121968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.121935    4810 cpu_manager.go:414] "containerMap: removing container" podUID="2d0e546b-643f-4f91-8060-30b6d1947681" containerName="payload-processing"
Oct 30 21:13:54.122069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.121969    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0e546b-643f-4f91-8060-30b6d1947681" containerName="payload-processing"
Oct 30 21:13:54.122069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.121989    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7158dcac-f804-4aaa-92bb-e6a83f5056b2" containerName="revoke-keys"
Oct 30 21:13:54.122069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.121997    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7158dcac-f804-4aaa-92bb-e6a83f5056b2" containerName="revoke-keys"
Oct 30 21:13:54.122278 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.122259    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0e546b-643f-4f91-8060-30b6d1947681" containerName="payload-processing"
Oct 30 21:13:54.122362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.122286    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7158dcac-f804-4aaa-92bb-e6a83f5056b2" containerName="revoke-keys"
Oct 30 21:13:54.127658 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.127635    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.134174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.134143    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"]
Oct 30 21:13:54.135229 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.135008    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:54.138345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.135586    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:13:54.154084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.154058    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbc49\" (UniqueName: \"kubernetes.io/projected/4b1739f7-ed64-45a4-bddf-955945c8d139-kube-api-access-gbc49\") pod \"maas-api-revoke-sub-3e48c045-z97rc\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.154420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.154396    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e48c045-z97rc\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.154534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.154456    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e48c045-z97rc\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.255364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.255335    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e48c045-z97rc\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.255517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.255436    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbc49\" (UniqueName: \"kubernetes.io/projected/4b1739f7-ed64-45a4-bddf-955945c8d139-kube-api-access-gbc49\") pod \"maas-api-revoke-sub-3e48c045-z97rc\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.255575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.255561    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e48c045-z97rc\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.256100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.256073    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e48c045-z97rc\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.258010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.257987    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e48c045-z97rc\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.264402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.264381    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbc49\" (UniqueName: \"kubernetes.io/projected/4b1739f7-ed64-45a4-bddf-955945c8d139-kube-api-access-gbc49\") pod \"maas-api-revoke-sub-3e48c045-z97rc\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.454376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.454346    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:54.746803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:54.746771    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"]
Oct 30 21:13:54.747998 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:13:54.747968    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1739f7_ed64_45a4_bddf_955945c8d139.slice/crio-78946d34207d9f46e7cd66cfeec4f229b3d05e17dde2b05b32f4e1c4dfebc6ef WatchSource:0}: Error finding container 78946d34207d9f46e7cd66cfeec4f229b3d05e17dde2b05b32f4e1c4dfebc6ef: Status 404 returned error can't find the container with id 78946d34207d9f46e7cd66cfeec4f229b3d05e17dde2b05b32f4e1c4dfebc6ef
Oct 30 21:13:55.172612 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:13:55.172565    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:13:55.203638 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:55.203605    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc" event={"ID":"4b1739f7-ed64-45a4-bddf-955945c8d139","Type":"ContainerStarted","Data":"78946d34207d9f46e7cd66cfeec4f229b3d05e17dde2b05b32f4e1c4dfebc6ef"}
Oct 30 21:13:56.212599 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:56.212563    4810 generic.go:356] "Generic (PLEG): container finished" podID="4b1739f7-ed64-45a4-bddf-955945c8d139" containerID="4d8462f23ac304bd191f2b8f1f95ca28253bed9c0587041f6ca6fa03132af329" exitCode=0
Oct 30 21:13:56.213140 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:56.212649    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc" event={"ID":"4b1739f7-ed64-45a4-bddf-955945c8d139","Type":"ContainerDied","Data":"4d8462f23ac304bd191f2b8f1f95ca28253bed9c0587041f6ca6fa03132af329"}
Oct 30 21:13:57.387127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:57.387095    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:57.499025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:57.498939    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1739f7-ed64-45a4-bddf-955945c8d139-kube-api-access-gbc49\" (UniqueName: \"kubernetes.io/projected/4b1739f7-ed64-45a4-bddf-955945c8d139-kube-api-access-gbc49\") pod \"4b1739f7-ed64-45a4-bddf-955945c8d139\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") "
Oct 30 21:13:57.499025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:57.499003    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-xks-ca\") pod \"4b1739f7-ed64-45a4-bddf-955945c8d139\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") "
Oct 30 21:13:57.499260 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:57.499236    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-openshift-ca\") pod \"4b1739f7-ed64-45a4-bddf-955945c8d139\" (UID: \"4b1739f7-ed64-45a4-bddf-955945c8d139\") "
Oct 30 21:13:57.499572 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:57.499549    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-openshift-ca" pod "4b1739f7-ed64-45a4-bddf-955945c8d139" (UID: "4b1739f7-ed64-45a4-bddf-955945c8d139"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:57.500241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:57.500217    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:57.501506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:57.501474    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1739f7-ed64-45a4-bddf-955945c8d139-kube-api-access-gbc49" pod "4b1739f7-ed64-45a4-bddf-955945c8d139" (UID: "4b1739f7-ed64-45a4-bddf-955945c8d139"). InnerVolumeSpecName "kube-api-access-gbc49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:57.501628 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:57.501538    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-xks-ca" pod "4b1739f7-ed64-45a4-bddf-955945c8d139" (UID: "4b1739f7-ed64-45a4-bddf-955945c8d139"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:57.602221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:57.602187    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbc49\" (UniqueName: \"kubernetes.io/projected/4b1739f7-ed64-45a4-bddf-955945c8d139-kube-api-access-gbc49\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:57.602221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:57.602218    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1739f7-ed64-45a4-bddf-955945c8d139-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:58.231089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:58.231056    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"
Oct 30 21:13:58.231258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:58.231088    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc" event={"ID":"4b1739f7-ed64-45a4-bddf-955945c8d139","Type":"ContainerDied","Data":"78946d34207d9f46e7cd66cfeec4f229b3d05e17dde2b05b32f4e1c4dfebc6ef"}
Oct 30 21:13:58.231258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:13:58.231138    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78946d34207d9f46e7cd66cfeec4f229b3d05e17dde2b05b32f4e1c4dfebc6ef"
Oct 30 21:14:05.201063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:05.201014    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"]
Oct 30 21:14:09.898924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:09.896898    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"]
Oct 30 21:14:09.898924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:09.898320    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4b1739f7-ed64-45a4-bddf-955945c8d139" containerName="revoke-keys"
Oct 30 21:14:09.898924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:09.898342    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1739f7-ed64-45a4-bddf-955945c8d139" containerName="revoke-keys"
Oct 30 21:14:09.898924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:09.898577    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1739f7-ed64-45a4-bddf-955945c8d139" containerName="revoke-keys"
Oct 30 21:14:09.928880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:09.924137    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:09.940910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:09.930642    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"agc-cred-a082f645dd4a\""
Oct 30 21:14:09.940910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:09.930724    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"]
Oct 30 21:14:10.049952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.031698    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.053554 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.053520    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcbl\" (UniqueName: \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-kube-api-access-zhcbl\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.054014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.053992    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.054507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.054476    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.156513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.156430    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.156513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.156470    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcbl\" (UniqueName: \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-kube-api-access-zhcbl\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.156774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.156527    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.156774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.156653    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.157330 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.157306    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.157485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.157457    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.159411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.159389    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.166081 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.166042    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcbl\" (UniqueName: \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-kube-api-access-zhcbl\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.201509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.201467    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq" podUID="9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb" containerName="payload-processing" containerID="cri-o://e42ecfd52d2e7348948cec99a167b29bafa5960ad84a37d96abd345eb2108c38" gracePeriod=25
Oct 30 21:14:10.303389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.303354    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:10.389748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.389524    4810 generic.go:356] "Generic (PLEG): container finished" podID="9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb" containerID="e42ecfd52d2e7348948cec99a167b29bafa5960ad84a37d96abd345eb2108c38" exitCode=0
Oct 30 21:14:10.389748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.389693    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq" event={"ID":"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb","Type":"ContainerDied","Data":"e42ecfd52d2e7348948cec99a167b29bafa5960ad84a37d96abd345eb2108c38"}
Oct 30 21:14:10.520809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.520457    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:14:10.561150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.561117    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-routing-overlay-volume\") pod \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") "
Oct 30 21:14:10.561387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.561356    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-provider-credentials\") pod \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") "
Oct 30 21:14:10.561507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.561453    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-kube-api-access-dw4zm\" (UniqueName: \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-kube-api-access-dw4zm\") pod \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") "
Oct 30 21:14:10.561507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.561502    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-plugins-config-volume\") pod \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\" (UID: \"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb\") "
Oct 30 21:14:10.561728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.561575    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-routing-overlay-volume" pod "9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb" (UID: "9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:10.562099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.562078    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:10.562312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.562286    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-plugins-config-volume" pod "9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb" (UID: "9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:10.564091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.564058    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-kube-api-access-dw4zm" pod "9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb" (UID: "9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb"). InnerVolumeSpecName "kube-api-access-dw4zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:10.564531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.564500    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-provider-credentials" pod "9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb" (UID: "9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:10.600003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.599967    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"]
Oct 30 21:14:10.663480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.663450    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:10.663583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.663486    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:10.663583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.663503    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw4zm\" (UniqueName: \"kubernetes.io/projected/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb-kube-api-access-dw4zm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:10.824261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.824172    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"]
Oct 30 21:14:10.862202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.862174    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"]
Oct 30 21:14:10.862978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.862961    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb" containerName="payload-processing"
Oct 30 21:14:10.863041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.862980    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb" containerName="payload-processing"
Oct 30 21:14:10.863166 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.863135    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb" containerName="payload-processing"
Oct 30 21:14:10.866997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.866978    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:10.873637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.873615    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"agc-cred-e324f53555e5\""
Oct 30 21:14:10.883492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.883465    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"]
Oct 30 21:14:10.968150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.968113    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:10.968502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.968242    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:10.968502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.968334    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2nv4\" (UniqueName: \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-kube-api-access-v2nv4\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:10.968502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:10.968385    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:11.069127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.069093    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:11.069292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.069191    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:11.069396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.069293    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2nv4\" (UniqueName: \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-kube-api-access-v2nv4\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:11.069396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.069330    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:11.069953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.069932    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:11.070047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.069990    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:11.072228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.072204    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:11.081239 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.081184    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2nv4\" (UniqueName: \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-kube-api-access-v2nv4\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:11.204935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.204897    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:11.401399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.401368    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"
Oct 30 21:14:11.401399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.401390    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq" event={"ID":"9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb","Type":"ContainerDied","Data":"140e1a7f0a215acba735d91f6a862910ebfd304a819860d0e50f33f515e34c9b"}
Oct 30 21:14:11.401666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.401462    4810 scope.go:122] "RemoveContainer" containerID="e42ecfd52d2e7348948cec99a167b29bafa5960ad84a37d96abd345eb2108c38"
Oct 30 21:14:11.404044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.403846    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj" event={"ID":"dc15f60e-e18e-400d-bddd-73a9f1f8c176","Type":"ContainerStarted","Data":"b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122"}
Oct 30 21:14:11.404195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.404089    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:11.404195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.404114    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj" event={"ID":"dc15f60e-e18e-400d-bddd-73a9f1f8c176","Type":"ContainerStarted","Data":"27aeff7120a85d78d58bdb035907232edc9794b92d769b4971994bf9061db11e"}
Oct 30 21:14:11.427396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.427357    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj" podStartSLOduration=2.427345133 podStartE2EDuration="2.427345133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:11.424279947 +0000 UTC m=+2217.916096507" watchObservedRunningTime="2026-10-30 21:14:11.427345133 +0000 UTC m=+2217.919161677"
Oct 30 21:14:11.442250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.442226    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"]
Oct 30 21:14:11.448740 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.448720    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a574c2/payload-processing-external-model-e2e-extmod-b-a574c2-f49cxt7vq"]
Oct 30 21:14:11.488792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.488764    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"]
Oct 30 21:14:11.490022 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:14:11.489989    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53419711_85c4_4dc5_93e0_524f8ff26222.slice/crio-77689f9dfa33d5005ad90b9f6118bf3d37741edf29e57100530add1d302ffdb7 WatchSource:0}: Error finding container 77689f9dfa33d5005ad90b9f6118bf3d37741edf29e57100530add1d302ffdb7: Status 404 returned error can't find the container with id 77689f9dfa33d5005ad90b9f6118bf3d37741edf29e57100530add1d302ffdb7
Oct 30 21:14:11.664536 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:11.664474    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb" path="/var/lib/kubelet/pods/9d0e9c8b-42b5-4258-ab33-d91a39d8c3cb/volumes"
Oct 30 21:14:12.414403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:12.414362    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh" event={"ID":"53419711-85c4-4dc5-93e0-524f8ff26222","Type":"ContainerStarted","Data":"dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea"}
Oct 30 21:14:12.414922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:12.414449    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh" event={"ID":"53419711-85c4-4dc5-93e0-524f8ff26222","Type":"ContainerStarted","Data":"77689f9dfa33d5005ad90b9f6118bf3d37741edf29e57100530add1d302ffdb7"}
Oct 30 21:14:12.414922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:12.414677    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:12.437241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:12.437199    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh" podStartSLOduration=2.437186518 podStartE2EDuration="2.437186518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:12.433396394 +0000 UTC m=+2218.925212943" watchObservedRunningTime="2026-10-30 21:14:12.437186518 +0000 UTC m=+2218.929003097"
Oct 30 21:14:16.898908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:16.898875    4810 scope.go:122] "RemoveContainer" containerID="dfe4878f7dd9845c5adead19781b537858515a4593cc14dd8f0dd49eb8661cdb"
Oct 30 21:14:16.910800 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:16.910769    4810 scope.go:122] "RemoveContainer" containerID="5849d8e720e935543a2eaa31cb3abbfc1d3ba002c226f9efe19c08ac867c07b8"
Oct 30 21:14:16.921113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:16.921090    4810 scope.go:122] "RemoveContainer" containerID="6e5eb7d9ee1c53297d0c2e90c5fd26df681775d9360eecaa02d2e3bc7890c3d7"
Oct 30 21:14:16.931051 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:16.931031    4810 scope.go:122] "RemoveContainer" containerID="b63b418e26361cbe56565ec73b8d4566ed3320309667503f99392bb7663d770a"
Oct 30 21:14:16.941008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:16.940989    4810 scope.go:122] "RemoveContainer" containerID="352d0289bf108964c65ecb1e924d7fa29d304da08899b8fb61f86d105b705775"
Oct 30 21:14:17.416625 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.416583    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj" podUID="dc15f60e-e18e-400d-bddd-73a9f1f8c176" containerName="payload-processing" containerID="cri-o://b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122" gracePeriod=25
Oct 30 21:14:17.713251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.713224    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:17.778085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.778037    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-routing-overlay-volume\") pod \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") "
Oct 30 21:14:17.778289 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.778263    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-kube-api-access-zhcbl\" (UniqueName: \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-kube-api-access-zhcbl\") pod \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") "
Oct 30 21:14:17.778366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.778317    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-plugins-config-volume\") pod \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") "
Oct 30 21:14:17.778431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.778368    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-provider-credentials\") pod \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\" (UID: \"dc15f60e-e18e-400d-bddd-73a9f1f8c176\") "
Oct 30 21:14:17.779091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.779061    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-plugins-config-volume" pod "dc15f60e-e18e-400d-bddd-73a9f1f8c176" (UID: "dc15f60e-e18e-400d-bddd-73a9f1f8c176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:17.779929 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.779905    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-routing-overlay-volume" pod "dc15f60e-e18e-400d-bddd-73a9f1f8c176" (UID: "dc15f60e-e18e-400d-bddd-73a9f1f8c176"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:17.781157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.781124    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-kube-api-access-zhcbl" pod "dc15f60e-e18e-400d-bddd-73a9f1f8c176" (UID: "dc15f60e-e18e-400d-bddd-73a9f1f8c176"). InnerVolumeSpecName "kube-api-access-zhcbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:17.781835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.781808    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-provider-credentials" pod "dc15f60e-e18e-400d-bddd-73a9f1f8c176" (UID: "dc15f60e-e18e-400d-bddd-73a9f1f8c176"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:17.880565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.880520    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhcbl\" (UniqueName: \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-kube-api-access-zhcbl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:17.880565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.880557    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:17.880565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.880570    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc15f60e-e18e-400d-bddd-73a9f1f8c176-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:17.880795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:17.880583    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc15f60e-e18e-400d-bddd-73a9f1f8c176-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:18.481597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:18.481561    4810 generic.go:356] "Generic (PLEG): container finished" podID="dc15f60e-e18e-400d-bddd-73a9f1f8c176" containerID="b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122" exitCode=0
Oct 30 21:14:18.482088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:18.481629    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"
Oct 30 21:14:18.482088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:18.481696    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj" event={"ID":"dc15f60e-e18e-400d-bddd-73a9f1f8c176","Type":"ContainerDied","Data":"b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122"}
Oct 30 21:14:18.482088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:18.481778    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj" event={"ID":"dc15f60e-e18e-400d-bddd-73a9f1f8c176","Type":"ContainerDied","Data":"27aeff7120a85d78d58bdb035907232edc9794b92d769b4971994bf9061db11e"}
Oct 30 21:14:18.482088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:18.481807    4810 scope.go:122] "RemoveContainer" containerID="b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122"
Oct 30 21:14:18.494358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:18.494339    4810 scope.go:122] "RemoveContainer" containerID="b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122"
Oct 30 21:14:18.494659 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:14:18.494639    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122\": container with ID starting with b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122 not found: ID does not exist" containerID="b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122"
Oct 30 21:14:18.494706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:18.494668    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122"} err="failed to get container status \"b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122\": rpc error: code = NotFound desc = could not find container \"b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122\": container with ID starting with b78bde20895c2ce62ed8d50b689434aec4eba29da8dc30a9102d6817382c4122 not found: ID does not exist"
Oct 30 21:14:18.518178 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:18.518148    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"]
Oct 30 21:14:18.530306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:18.530282    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfm2csj"]
Oct 30 21:14:19.664096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:19.664062    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc15f60e-e18e-400d-bddd-73a9f1f8c176" path="/var/lib/kubelet/pods/dc15f60e-e18e-400d-bddd-73a9f1f8c176/volumes"
Oct 30 21:14:20.867161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:20.867132    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"]
Oct 30 21:14:20.867952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:20.867937    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dc15f60e-e18e-400d-bddd-73a9f1f8c176" containerName="payload-processing"
Oct 30 21:14:20.868005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:20.867953    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc15f60e-e18e-400d-bddd-73a9f1f8c176" containerName="payload-processing"
Oct 30 21:14:20.868071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:20.868063    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc15f60e-e18e-400d-bddd-73a9f1f8c176" containerName="payload-processing"
Oct 30 21:14:20.873228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:20.873208    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:20.879952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:20.879930    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:20.880390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:20.880370    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:14:20.886564 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:20.886542    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"]
Oct 30 21:14:21.032915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.032870    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dc17e63-dlxpq\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:21.033092 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.033066    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dc17e63-dlxpq\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:21.033134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.033103    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtssp\" (UniqueName: \"kubernetes.io/projected/7330b2a2-e159-4d20-9d40-d8da51079d24-kube-api-access-wtssp\") pod \"maas-api-revoke-sub-8dc17e63-dlxpq\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:21.135152 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.135114    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dc17e63-dlxpq\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:21.135323 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.135178    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtssp\" (UniqueName: \"kubernetes.io/projected/7330b2a2-e159-4d20-9d40-d8da51079d24-kube-api-access-wtssp\") pod \"maas-api-revoke-sub-8dc17e63-dlxpq\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:21.135411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.135387    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dc17e63-dlxpq\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:21.136245 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.136205    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dc17e63-dlxpq\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:21.138058 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.138033    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dc17e63-dlxpq\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:21.145316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.145291    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtssp\" (UniqueName: \"kubernetes.io/projected/7330b2a2-e159-4d20-9d40-d8da51079d24-kube-api-access-wtssp\") pod \"maas-api-revoke-sub-8dc17e63-dlxpq\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:21.200345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.200310    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:21.515019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:21.514988    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"]
Oct 30 21:14:21.515513 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:14:21.515483    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7330b2a2_e159_4d20_9d40_d8da51079d24.slice/crio-2c6ca8d35524e9c7f62a0f911867c8c3598cbc13454798a4dd2c082aad3c13ee WatchSource:0}: Error finding container 2c6ca8d35524e9c7f62a0f911867c8c3598cbc13454798a4dd2c082aad3c13ee: Status 404 returned error can't find the container with id 2c6ca8d35524e9c7f62a0f911867c8c3598cbc13454798a4dd2c082aad3c13ee
Oct 30 21:14:22.521141 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:22.521098    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq" event={"ID":"7330b2a2-e159-4d20-9d40-d8da51079d24","Type":"ContainerStarted","Data":"46bb17c2354a4e18dfef3881690916874621fa9ba2ae44920065c903d3bd1ea9"}
Oct 30 21:14:22.521624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:22.521157    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq" event={"ID":"7330b2a2-e159-4d20-9d40-d8da51079d24","Type":"ContainerStarted","Data":"2c6ca8d35524e9c7f62a0f911867c8c3598cbc13454798a4dd2c082aad3c13ee"}
Oct 30 21:14:22.575791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:22.575736    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq" podStartSLOduration=2.575711663 podStartE2EDuration="2.575711663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:22.574575193 +0000 UTC m=+2229.066391739" watchObservedRunningTime="2026-10-30 21:14:22.575711663 +0000 UTC m=+2229.067528209"
Oct 30 21:14:23.429926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.429894    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:14:23.531136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.531099    4810 generic.go:356] "Generic (PLEG): container finished" podID="7330b2a2-e159-4d20-9d40-d8da51079d24" containerID="46bb17c2354a4e18dfef3881690916874621fa9ba2ae44920065c903d3bd1ea9" exitCode=0
Oct 30 21:14:23.531594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.531154    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq" event={"ID":"7330b2a2-e159-4d20-9d40-d8da51079d24","Type":"ContainerDied","Data":"46bb17c2354a4e18dfef3881690916874621fa9ba2ae44920065c903d3bd1ea9"}
Oct 30 21:14:23.568300 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.568273    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"]
Oct 30 21:14:23.924687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.924645    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"]
Oct 30 21:14:23.929649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.929624    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:23.938301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.938277    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-345bc4\"/\"kube-root-ca.crt\""
Oct 30 21:14:23.938438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.938334    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-345bc4\"/\"agc-cred-f68932a2d373\""
Oct 30 21:14:23.938438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.938398    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-345bc4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-345bc4\""
Oct 30 21:14:23.938438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.938437    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-345bc4\"/\"routing-overlay\""
Oct 30 21:14:23.939127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.938789    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-345bc4\"/\"openshift-service-ca.crt\""
Oct 30 21:14:23.939127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.938958    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-345bc4\"/\"payload-processing-external-model-e2e--fae60aa2-dockercfg-7h2d7\""
Oct 30 21:14:23.944246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:23.944222    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"]
Oct 30 21:14:24.126889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.124025    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.126889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.124118    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.126889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.124298    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs922\" (UniqueName: \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-kube-api-access-xs922\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.126889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.124338    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.226366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.226263    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs922\" (UniqueName: \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-kube-api-access-xs922\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.226366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.226312    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.226613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.226412    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.226613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.226455    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.227044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.227017    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.227205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.227184    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.228943 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.228925    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.238795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.238769    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs922\" (UniqueName: \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-kube-api-access-xs922\") pod \"payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") " pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.246747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.246727    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:24.550118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.550092    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"]
Oct 30 21:14:24.552606 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:14:24.552580    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfbdf2a_9bad_4968_acb2_4759ca868815.slice/crio-6a3926f4f1a5e7b79f1ffda6629b9d94c98c449e2d130f8673b6845c003123de WatchSource:0}: Error finding container 6a3926f4f1a5e7b79f1ffda6629b9d94c98c449e2d130f8673b6845c003123de: Status 404 returned error can't find the container with id 6a3926f4f1a5e7b79f1ffda6629b9d94c98c449e2d130f8673b6845c003123de
Oct 30 21:14:24.723663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.723636    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:24.838817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.838746    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-openshift-ca\") pod \"7330b2a2-e159-4d20-9d40-d8da51079d24\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") "
Oct 30 21:14:24.838990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.838893    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-xks-ca\") pod \"7330b2a2-e159-4d20-9d40-d8da51079d24\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") "
Oct 30 21:14:24.839079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.838996    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7330b2a2-e159-4d20-9d40-d8da51079d24-kube-api-access-wtssp\" (UniqueName: \"kubernetes.io/projected/7330b2a2-e159-4d20-9d40-d8da51079d24-kube-api-access-wtssp\") pod \"7330b2a2-e159-4d20-9d40-d8da51079d24\" (UID: \"7330b2a2-e159-4d20-9d40-d8da51079d24\") "
Oct 30 21:14:24.839187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.839160    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-openshift-ca" pod "7330b2a2-e159-4d20-9d40-d8da51079d24" (UID: "7330b2a2-e159-4d20-9d40-d8da51079d24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:24.839818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.839772    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:24.841207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.841183    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-xks-ca" pod "7330b2a2-e159-4d20-9d40-d8da51079d24" (UID: "7330b2a2-e159-4d20-9d40-d8da51079d24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:24.841303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.841282    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7330b2a2-e159-4d20-9d40-d8da51079d24-kube-api-access-wtssp" pod "7330b2a2-e159-4d20-9d40-d8da51079d24" (UID: "7330b2a2-e159-4d20-9d40-d8da51079d24"). InnerVolumeSpecName "kube-api-access-wtssp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:24.941210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.941185    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtssp\" (UniqueName: \"kubernetes.io/projected/7330b2a2-e159-4d20-9d40-d8da51079d24-kube-api-access-wtssp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:24.941210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:24.941213    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7330b2a2-e159-4d20-9d40-d8da51079d24-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:25.156792 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:14:25.156759    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:14:25.548048 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:25.547975    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"
Oct 30 21:14:25.548048 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:25.548020    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq" event={"ID":"7330b2a2-e159-4d20-9d40-d8da51079d24","Type":"ContainerDied","Data":"2c6ca8d35524e9c7f62a0f911867c8c3598cbc13454798a4dd2c082aad3c13ee"}
Oct 30 21:14:25.548351 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:25.548079    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c6ca8d35524e9c7f62a0f911867c8c3598cbc13454798a4dd2c082aad3c13ee"
Oct 30 21:14:25.549903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:25.549846    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv" event={"ID":"1dfbdf2a-9bad-4968-acb2-4759ca868815","Type":"ContainerStarted","Data":"9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0"}
Oct 30 21:14:25.550062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:25.549922    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv" event={"ID":"1dfbdf2a-9bad-4968-acb2-4759ca868815","Type":"ContainerStarted","Data":"6a3926f4f1a5e7b79f1ffda6629b9d94c98c449e2d130f8673b6845c003123de"}
Oct 30 21:14:25.550062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:25.550052    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:25.572933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:25.572892    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv" podStartSLOduration=2.572842977 podStartE2EDuration="2.572842977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:25.570454062 +0000 UTC m=+2232.062270608" watchObservedRunningTime="2026-10-30 21:14:25.572842977 +0000 UTC m=+2232.064659521"
Oct 30 21:14:28.568831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.568788    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r" podUID="c5afc507-5879-48be-b497-34e36983685f" containerName="payload-processing" containerID="cri-o://51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1" gracePeriod=25
Oct 30 21:14:28.855695 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.855669    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:14:28.906650 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.906618    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-routing-overlay-volume\") pod \"c5afc507-5879-48be-b497-34e36983685f\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") "
Oct 30 21:14:28.906843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.906658    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-provider-credentials\") pod \"c5afc507-5879-48be-b497-34e36983685f\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") "
Oct 30 21:14:28.906843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.906697    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-plugins-config-volume\") pod \"c5afc507-5879-48be-b497-34e36983685f\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") "
Oct 30 21:14:28.907021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.906991    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-kube-api-access-bst79\" (UniqueName: \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-kube-api-access-bst79\") pod \"c5afc507-5879-48be-b497-34e36983685f\" (UID: \"c5afc507-5879-48be-b497-34e36983685f\") "
Oct 30 21:14:28.907492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.907455    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-plugins-config-volume" pod "c5afc507-5879-48be-b497-34e36983685f" (UID: "c5afc507-5879-48be-b497-34e36983685f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:28.907687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.907666    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:28.908034 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.908006    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-routing-overlay-volume" pod "c5afc507-5879-48be-b497-34e36983685f" (UID: "c5afc507-5879-48be-b497-34e36983685f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:28.916068 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.916040    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-kube-api-access-bst79" pod "c5afc507-5879-48be-b497-34e36983685f" (UID: "c5afc507-5879-48be-b497-34e36983685f"). InnerVolumeSpecName "kube-api-access-bst79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:28.916173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:28.916079    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-provider-credentials" pod "c5afc507-5879-48be-b497-34e36983685f" (UID: "c5afc507-5879-48be-b497-34e36983685f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:29.008975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.008935    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bst79\" (UniqueName: \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-kube-api-access-bst79\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:29.008975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.008965    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5afc507-5879-48be-b497-34e36983685f-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:29.008975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.008974    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5afc507-5879-48be-b497-34e36983685f-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:29.589397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.589354    4810 generic.go:356] "Generic (PLEG): container finished" podID="c5afc507-5879-48be-b497-34e36983685f" containerID="51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1" exitCode=0
Oct 30 21:14:29.589933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.589404    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"
Oct 30 21:14:29.589933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.589432    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r" event={"ID":"c5afc507-5879-48be-b497-34e36983685f","Type":"ContainerDied","Data":"51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1"}
Oct 30 21:14:29.589933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.589499    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r" event={"ID":"c5afc507-5879-48be-b497-34e36983685f","Type":"ContainerDied","Data":"9d62c2cb51ee9997c15154b5226288e25f6d9cb35d121d6d6702b33977cbb471"}
Oct 30 21:14:29.589933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.589526    4810 scope.go:122] "RemoveContainer" containerID="51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1"
Oct 30 21:14:29.601811 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.601793    4810 scope.go:122] "RemoveContainer" containerID="51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1"
Oct 30 21:14:29.602141 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:14:29.602111    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1\": container with ID starting with 51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1 not found: ID does not exist" containerID="51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1"
Oct 30 21:14:29.602271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.602149    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1"} err="failed to get container status \"51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1\": rpc error: code = NotFound desc = could not find container \"51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1\": container with ID starting with 51eff732d8763096428231798ad79a49ac620fece56434d31334f3dcb71448f1 not found: ID does not exist"
Oct 30 21:14:29.618343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.618307    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"]
Oct 30 21:14:29.622379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.622359    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f4952cw4r"]
Oct 30 21:14:29.663617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:29.663595    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5afc507-5879-48be-b497-34e36983685f" path="/var/lib/kubelet/pods/c5afc507-5879-48be-b497-34e36983685f/volumes"
Oct 30 21:14:31.962527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:31.962492    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"]
Oct 30 21:14:35.238217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.238181    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"]
Oct 30 21:14:35.239035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.239010    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7330b2a2-e159-4d20-9d40-d8da51079d24" containerName="revoke-keys"
Oct 30 21:14:35.239035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.239033    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7330b2a2-e159-4d20-9d40-d8da51079d24" containerName="revoke-keys"
Oct 30 21:14:35.239203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.239065    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c5afc507-5879-48be-b497-34e36983685f" containerName="payload-processing"
Oct 30 21:14:35.239203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.239078    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5afc507-5879-48be-b497-34e36983685f" containerName="payload-processing"
Oct 30 21:14:35.239281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.239269    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7330b2a2-e159-4d20-9d40-d8da51079d24" containerName="revoke-keys"
Oct 30 21:14:35.239316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.239284    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5afc507-5879-48be-b497-34e36983685f" containerName="payload-processing"
Oct 30 21:14:35.245017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.244996    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.249665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.249619    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-00909f\"/\"agc-cred-12c5bf10925d\""
Oct 30 21:14:35.249812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.249706    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-00909f\"/\"payload-processing-external-model-e2e--3c4e5353-dockercfg-glf57\""
Oct 30 21:14:35.254079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.253190    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"]
Oct 30 21:14:35.346444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.346409    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.346701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.346624    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.346911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.346700    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdf6f\" (UniqueName: \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-kube-api-access-xdf6f\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.346911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.346745    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.450903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.448385    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.450903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.448666    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.450903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.448746    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdf6f\" (UniqueName: \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-kube-api-access-xdf6f\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.450903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.448789    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.450903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.449495    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.450903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.450016    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.453361 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.453331    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.459944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.459912    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdf6f\" (UniqueName: \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-kube-api-access-xdf6f\") pod \"payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") " pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.563079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.562980    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:35.895971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:35.895936    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"]
Oct 30 21:14:35.900284 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:14:35.900244    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bac0e38_3d1a_438d_ae21_1186242471b9.slice/crio-e9518da2ae9fc57b779551361b5628a1e38f6d61fc39a18d0f5074d7955729cc WatchSource:0}: Error finding container e9518da2ae9fc57b779551361b5628a1e38f6d61fc39a18d0f5074d7955729cc: Status 404 returned error can't find the container with id e9518da2ae9fc57b779551361b5628a1e38f6d61fc39a18d0f5074d7955729cc
Oct 30 21:14:36.563169 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:36.563140    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:14:36.703833 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:36.702609    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp" event={"ID":"1bac0e38-3d1a-438d-ae21-1186242471b9","Type":"ContainerStarted","Data":"ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c"}
Oct 30 21:14:36.703833 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:36.702683    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp" event={"ID":"1bac0e38-3d1a-438d-ae21-1186242471b9","Type":"ContainerStarted","Data":"e9518da2ae9fc57b779551361b5628a1e38f6d61fc39a18d0f5074d7955729cc"}
Oct 30 21:14:36.703833 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:36.702934    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:36.722681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:36.722628    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp" podStartSLOduration=1.7226141529999999 podStartE2EDuration="1.722614153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:36.720174489 +0000 UTC m=+2243.211991033" watchObservedRunningTime="2026-10-30 21:14:36.722614153 +0000 UTC m=+2243.214430697"
Oct 30 21:14:36.963331 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:36.963286    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv" podUID="68c0e156-6da3-4e3d-afe1-bd28d029505c" containerName="payload-processing" containerID="cri-o://fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f" gracePeriod=25
Oct 30 21:14:37.265754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.265727    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:14:37.364130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.364100    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:14:37.376896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.376852    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-routing-overlay-volume\") pod \"68c0e156-6da3-4e3d-afe1-bd28d029505c\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") "
Oct 30 21:14:37.377055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.376935    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-provider-credentials\") pod \"68c0e156-6da3-4e3d-afe1-bd28d029505c\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") "
Oct 30 21:14:37.377133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.377100    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-kube-api-access-dwhtn\" (UniqueName: \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-kube-api-access-dwhtn\") pod \"68c0e156-6da3-4e3d-afe1-bd28d029505c\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") "
Oct 30 21:14:37.377240 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.377222    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-plugins-config-volume\") pod \"68c0e156-6da3-4e3d-afe1-bd28d029505c\" (UID: \"68c0e156-6da3-4e3d-afe1-bd28d029505c\") "
Oct 30 21:14:37.377340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.377313    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-routing-overlay-volume" pod "68c0e156-6da3-4e3d-afe1-bd28d029505c" (UID: "68c0e156-6da3-4e3d-afe1-bd28d029505c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:37.377618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.377601    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-plugins-config-volume" pod "68c0e156-6da3-4e3d-afe1-bd28d029505c" (UID: "68c0e156-6da3-4e3d-afe1-bd28d029505c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:37.377839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.377816    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:37.377942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.377846    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68c0e156-6da3-4e3d-afe1-bd28d029505c-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:37.379339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.379315    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-provider-credentials" pod "68c0e156-6da3-4e3d-afe1-bd28d029505c" (UID: "68c0e156-6da3-4e3d-afe1-bd28d029505c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:37.379627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.379603    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-kube-api-access-dwhtn" pod "68c0e156-6da3-4e3d-afe1-bd28d029505c" (UID: "68c0e156-6da3-4e3d-afe1-bd28d029505c"). InnerVolumeSpecName "kube-api-access-dwhtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:37.478846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.478814    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhtn\" (UniqueName: \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-kube-api-access-dwhtn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:37.478846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.478842    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68c0e156-6da3-4e3d-afe1-bd28d029505c-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:37.713810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.713775    4810 generic.go:356] "Generic (PLEG): container finished" podID="68c0e156-6da3-4e3d-afe1-bd28d029505c" containerID="fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f" exitCode=0
Oct 30 21:14:37.714267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.713835    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"
Oct 30 21:14:37.714267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.713892    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv" event={"ID":"68c0e156-6da3-4e3d-afe1-bd28d029505c","Type":"ContainerDied","Data":"fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f"}
Oct 30 21:14:37.714267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.713965    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv" event={"ID":"68c0e156-6da3-4e3d-afe1-bd28d029505c","Type":"ContainerDied","Data":"c7f9d57f2835a700cbd402ad6a91d977e58ed29c9a6fdd0aed7454f08cbffb50"}
Oct 30 21:14:37.714267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.713989    4810 scope.go:122] "RemoveContainer" containerID="fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f"
Oct 30 21:14:37.725194 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.725176    4810 scope.go:122] "RemoveContainer" containerID="fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f"
Oct 30 21:14:37.725497 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:14:37.725466    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f\": container with ID starting with fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f not found: ID does not exist" containerID="fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f"
Oct 30 21:14:37.725590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.725500    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f"} err="failed to get container status \"fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f\": rpc error: code = NotFound desc = could not find container \"fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f\": container with ID starting with fabbdbdc533eb81a79d169b8f7d7e24b6283a4dd936de8674e6ec0694174148f not found: ID does not exist"
Oct 30 21:14:37.742092 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.742069    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"]
Oct 30 21:14:37.746369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:37.746349    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-67f4lkpwv"]
Oct 30 21:14:39.664783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:39.664754    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c0e156-6da3-4e3d-afe1-bd28d029505c" path="/var/lib/kubelet/pods/68c0e156-6da3-4e3d-afe1-bd28d029505c/volumes"
Oct 30 21:14:42.216896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.213028    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"]
Oct 30 21:14:42.216896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.215224    4810 cpu_manager.go:414] "containerMap: removing container" podUID="68c0e156-6da3-4e3d-afe1-bd28d029505c" containerName="payload-processing"
Oct 30 21:14:42.216896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.215248    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c0e156-6da3-4e3d-afe1-bd28d029505c" containerName="payload-processing"
Oct 30 21:14:42.216896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.215896    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c0e156-6da3-4e3d-afe1-bd28d029505c" containerName="payload-processing"
Oct 30 21:14:42.230550 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.230510    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.235319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.235288    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"agc-cred-420cdde55a4d\""
Oct 30 21:14:42.239182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.239151    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"]
Oct 30 21:14:42.352733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.352696    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.352932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.352743    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.353086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.353059    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.353128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.353115    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8722b\" (UniqueName: \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-kube-api-access-8722b\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.454746 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.454716    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.454938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.454757    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.455006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.454942    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.455006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.454962    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8722b\" (UniqueName: \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-kube-api-access-8722b\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.455418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.455391    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.455549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.455530    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.457721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.457697    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.464922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.464896    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8722b\" (UniqueName: \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-kube-api-access-8722b\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.554264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.554195    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:42.844244 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:14:42.844198    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19367ab4_c5b3_4f02_bc95_760f48d1292c.slice/crio-6b959d30116bcf89c8ba4fe49be522021ea00beb415b2c8c65f963cd7d662730 WatchSource:0}: Error finding container 6b959d30116bcf89c8ba4fe49be522021ea00beb415b2c8c65f963cd7d662730: Status 404 returned error can't find the container with id 6b959d30116bcf89c8ba4fe49be522021ea00beb415b2c8c65f963cd7d662730
Oct 30 21:14:42.845347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:42.845324    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"]
Oct 30 21:14:43.807428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:43.807390    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f" event={"ID":"19367ab4-c5b3-4f02-bc95-760f48d1292c","Type":"ContainerStarted","Data":"a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c"}
Oct 30 21:14:43.807946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:43.807456    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f" event={"ID":"19367ab4-c5b3-4f02-bc95-760f48d1292c","Type":"ContainerStarted","Data":"6b959d30116bcf89c8ba4fe49be522021ea00beb415b2c8c65f963cd7d662730"}
Oct 30 21:14:43.807946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:43.807488    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:43.827440 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:43.827388    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f" podStartSLOduration=1.827374796 podStartE2EDuration="1.827374796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:43.825089901 +0000 UTC m=+2250.316906446" watchObservedRunningTime="2026-10-30 21:14:43.827374796 +0000 UTC m=+2250.319191341"
Oct 30 21:14:47.718077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:47.718044    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:14:54.821934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:54.821901    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:14:54.879391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:54.879355    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"]
Oct 30 21:14:55.159003 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:14:55.158967    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:14:59.880580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:14:59.880527    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97" podUID="a5b886ae-5b1c-43fb-94ff-92c7ac1f3683" containerName="payload-processing" containerID="cri-o://ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621" gracePeriod=25
Oct 30 21:15:00.166904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.166846    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"]
Oct 30 21:15:00.172430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.172408    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"]
Oct 30 21:15:00.172557 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.172541    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"
Oct 30 21:15:00.194614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.184250    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:15:00.194614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.185126    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:15:00.199077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.194986    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"]
Oct 30 21:15:00.199077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.195088    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"
Oct 30 21:15:00.199344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.199319    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a5b886ae-5b1c-43fb-94ff-92c7ac1f3683" containerName="payload-processing"
Oct 30 21:15:00.199403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.199347    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b886ae-5b1c-43fb-94ff-92c7ac1f3683" containerName="payload-processing"
Oct 30 21:15:00.199602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.199586    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b886ae-5b1c-43fb-94ff-92c7ac1f3683" containerName="payload-processing"
Oct 30 21:15:00.230709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.226004    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"]
Oct 30 21:15:00.233915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.231528    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"
Oct 30 21:15:00.244057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.242669    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"]
Oct 30 21:15:00.244057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.242847    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"
Oct 30 21:15:00.257900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.257055    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"]
Oct 30 21:15:00.257900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.257094    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"]
Oct 30 21:15:00.257900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.257165    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"
Oct 30 21:15:00.262000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.261964    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"]
Oct 30 21:15:00.266300 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.266274    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"]
Oct 30 21:15:00.275358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.274990    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"]
Oct 30 21:15:00.277415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.276292    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"
Oct 30 21:15:00.292633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.292596    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"]
Oct 30 21:15:00.292776 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.292643    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"]
Oct 30 21:15:00.292776 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.292729    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.317073    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-plugins-config-volume\") pod \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") "
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.317178    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-routing-overlay-volume\") pod \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") "
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.317235    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-kube-api-access-r4gps\" (UniqueName: \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-kube-api-access-r4gps\") pod \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") "
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.317308    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-provider-credentials\") pod \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\" (UID: \"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683\") "
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.317727    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e47096-ee02-476d-ab4c-d7d0846a976a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n\" (UID: \"a0e47096-ee02-476d-ab4c-d7d0846a976a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.317784    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467ll\" (UniqueName: \"kubernetes.io/projected/2d1184f0-d306-4de0-a4f3-016c48e5f897-kube-api-access-467ll\") pod \"maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g\" (UID: \"2d1184f0-d306-4de0-a4f3-016c48e5f897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.317826    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxgq\" (UniqueName: \"kubernetes.io/projected/a0e47096-ee02-476d-ab4c-d7d0846a976a-kube-api-access-6bxgq\") pod \"maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n\" (UID: \"a0e47096-ee02-476d-ab4c-d7d0846a976a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.317961    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq879\" (UniqueName: \"kubernetes.io/projected/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-kube-api-access-hq879\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm\" (UID: \"fb4d6041-3a3c-4980-9f2f-849b4bb7d457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.318030    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def80bf3-e036-460e-9543-114a136c81fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts\" (UID: \"def80bf3-e036-460e-9543-114a136c81fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.318093    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm\" (UID: \"fb4d6041-3a3c-4980-9f2f-849b4bb7d457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.318122    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1184f0-d306-4de0-a4f3-016c48e5f897-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g\" (UID: \"2d1184f0-d306-4de0-a4f3-016c48e5f897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.318189    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtr49\" (UniqueName: \"kubernetes.io/projected/def80bf3-e036-460e-9543-114a136c81fe-kube-api-access-dtr49\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts\" (UID: \"def80bf3-e036-460e-9543-114a136c81fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.318238    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7pxg\" (UniqueName: \"kubernetes.io/projected/18183221-a049-4171-add0-1810e4939630-kube-api-access-t7pxg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629\" (UID: \"18183221-a049-4171-add0-1810e4939630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"
Oct 30 21:15:00.318431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.318277    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18183221-a049-4171-add0-1810e4939630-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629\" (UID: \"18183221-a049-4171-add0-1810e4939630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"
Oct 30 21:15:00.319224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.318780    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-plugins-config-volume" pod "a5b886ae-5b1c-43fb-94ff-92c7ac1f3683" (UID: "a5b886ae-5b1c-43fb-94ff-92c7ac1f3683"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:00.319224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.319165    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-routing-overlay-volume" pod "a5b886ae-5b1c-43fb-94ff-92c7ac1f3683" (UID: "a5b886ae-5b1c-43fb-94ff-92c7ac1f3683"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:00.339256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.336795    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"]
Oct 30 21:15:00.339256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.337593    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-kube-api-access-r4gps" pod "a5b886ae-5b1c-43fb-94ff-92c7ac1f3683" (UID: "a5b886ae-5b1c-43fb-94ff-92c7ac1f3683"). InnerVolumeSpecName "kube-api-access-r4gps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:00.339256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.338682    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-provider-credentials" pod "a5b886ae-5b1c-43fb-94ff-92c7ac1f3683" (UID: "a5b886ae-5b1c-43fb-94ff-92c7ac1f3683"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:00.348118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.346109    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"]
Oct 30 21:15:00.353766 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.353742    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"
Oct 30 21:15:00.356769 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.356670    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"]
Oct 30 21:15:00.361626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.361604    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"]
Oct 30 21:15:00.367835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.367809    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"
Oct 30 21:15:00.371467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.371447    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"]
Oct 30 21:15:00.384928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.384911    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"
Oct 30 21:15:00.387373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.387319    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"]
Oct 30 21:15:00.410794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.410606    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"]
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.419653    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqc6c\" (UniqueName: \"kubernetes.io/projected/c29ca7de-8f46-4550-9e3b-58e215775d2d-kube-api-access-zqc6c\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24\" (UID: \"c29ca7de-8f46-4550-9e3b-58e215775d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.419717    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e47096-ee02-476d-ab4c-d7d0846a976a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n\" (UID: \"a0e47096-ee02-476d-ab4c-d7d0846a976a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.419798    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe530dfb-4534-49d6-91d3-28f198c77d88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66\" (UID: \"fe530dfb-4534-49d6-91d3-28f198c77d88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.419826    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91a2f39-f9cc-4bd9-8794-98d575ca8481-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q\" (UID: \"f91a2f39-f9cc-4bd9-8794-98d575ca8481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.419875    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-467ll\" (UniqueName: \"kubernetes.io/projected/2d1184f0-d306-4de0-a4f3-016c48e5f897-kube-api-access-467ll\") pod \"maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g\" (UID: \"2d1184f0-d306-4de0-a4f3-016c48e5f897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.419924    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxgq\" (UniqueName: \"kubernetes.io/projected/a0e47096-ee02-476d-ab4c-d7d0846a976a-kube-api-access-6bxgq\") pod \"maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n\" (UID: \"a0e47096-ee02-476d-ab4c-d7d0846a976a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.419975    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq879\" (UniqueName: \"kubernetes.io/projected/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-kube-api-access-hq879\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm\" (UID: \"fb4d6041-3a3c-4980-9f2f-849b4bb7d457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.420033    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"]
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.420747    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def80bf3-e036-460e-9543-114a136c81fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts\" (UID: \"def80bf3-e036-460e-9543-114a136c81fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.421445    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e47096-ee02-476d-ab4c-d7d0846a976a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n\" (UID: \"a0e47096-ee02-476d-ab4c-d7d0846a976a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.421933    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"
Oct 30 21:15:00.422123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.420035    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def80bf3-e036-460e-9543-114a136c81fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts\" (UID: \"def80bf3-e036-460e-9543-114a136c81fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422151    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29ca7de-8f46-4550-9e3b-58e215775d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24\" (UID: \"c29ca7de-8f46-4550-9e3b-58e215775d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422176    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbsv2\" (UniqueName: \"kubernetes.io/projected/f91a2f39-f9cc-4bd9-8794-98d575ca8481-kube-api-access-sbsv2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q\" (UID: \"f91a2f39-f9cc-4bd9-8794-98d575ca8481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422237    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm\" (UID: \"fb4d6041-3a3c-4980-9f2f-849b4bb7d457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422294    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1184f0-d306-4de0-a4f3-016c48e5f897-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g\" (UID: \"2d1184f0-d306-4de0-a4f3-016c48e5f897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422361    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtr49\" (UniqueName: \"kubernetes.io/projected/def80bf3-e036-460e-9543-114a136c81fe-kube-api-access-dtr49\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts\" (UID: \"def80bf3-e036-460e-9543-114a136c81fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422409    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7pxg\" (UniqueName: \"kubernetes.io/projected/18183221-a049-4171-add0-1810e4939630-kube-api-access-t7pxg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629\" (UID: \"18183221-a049-4171-add0-1810e4939630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422442    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4d5\" (UniqueName: \"kubernetes.io/projected/fe530dfb-4534-49d6-91d3-28f198c77d88-kube-api-access-kt4d5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66\" (UID: \"fe530dfb-4534-49d6-91d3-28f198c77d88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422474    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18183221-a049-4171-add0-1810e4939630-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629\" (UID: \"18183221-a049-4171-add0-1810e4939630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422638    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422655    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422669    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4gps\" (UniqueName: \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-kube-api-access-r4gps\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:00.422921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.422684    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:00.423576 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.423364    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18183221-a049-4171-add0-1810e4939630-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629\" (UID: \"18183221-a049-4171-add0-1810e4939630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"
Oct 30 21:15:00.424109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.424082    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm\" (UID: \"fb4d6041-3a3c-4980-9f2f-849b4bb7d457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"
Oct 30 21:15:00.424745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.424723    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1184f0-d306-4de0-a4f3-016c48e5f897-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g\" (UID: \"2d1184f0-d306-4de0-a4f3-016c48e5f897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"
Oct 30 21:15:00.432206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.432185    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"]
Oct 30 21:15:00.432311 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.432293    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"
Oct 30 21:15:00.433561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.433275    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtr49\" (UniqueName: \"kubernetes.io/projected/def80bf3-e036-460e-9543-114a136c81fe-kube-api-access-dtr49\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts\" (UID: \"def80bf3-e036-460e-9543-114a136c81fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"
Oct 30 21:15:00.433561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.433463    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-467ll\" (UniqueName: \"kubernetes.io/projected/2d1184f0-d306-4de0-a4f3-016c48e5f897-kube-api-access-467ll\") pod \"maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g\" (UID: \"2d1184f0-d306-4de0-a4f3-016c48e5f897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"
Oct 30 21:15:00.433561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.433509    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"]
Oct 30 21:15:00.434646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.434548    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxgq\" (UniqueName: \"kubernetes.io/projected/a0e47096-ee02-476d-ab4c-d7d0846a976a-kube-api-access-6bxgq\") pod \"maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n\" (UID: \"a0e47096-ee02-476d-ab4c-d7d0846a976a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"
Oct 30 21:15:00.434748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.434704    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq879\" (UniqueName: \"kubernetes.io/projected/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-kube-api-access-hq879\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm\" (UID: \"fb4d6041-3a3c-4980-9f2f-849b4bb7d457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"
Oct 30 21:15:00.437930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.437267    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7pxg\" (UniqueName: \"kubernetes.io/projected/18183221-a049-4171-add0-1810e4939630-kube-api-access-t7pxg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629\" (UID: \"18183221-a049-4171-add0-1810e4939630\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"
Oct 30 21:15:00.439878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.439686    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"]
Oct 30 21:15:00.443554 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.443529    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"]
Oct 30 21:15:00.448354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.448336    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"
Oct 30 21:15:00.455716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.452571    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"]
Oct 30 21:15:00.458663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.458642    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"
Oct 30 21:15:00.459598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.459573    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"]
Oct 30 21:15:00.465511 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.465489    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"]
Oct 30 21:15:00.465612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.465591    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"
Oct 30 21:15:00.472559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.472535    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"]
Oct 30 21:15:00.482622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.482600    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"]
Oct 30 21:15:00.493213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.493195    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"]
Oct 30 21:15:00.524597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.524562    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqc6c\" (UniqueName: \"kubernetes.io/projected/c29ca7de-8f46-4550-9e3b-58e215775d2d-kube-api-access-zqc6c\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24\" (UID: \"c29ca7de-8f46-4550-9e3b-58e215775d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"
Oct 30 21:15:00.524752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.524654    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe530dfb-4534-49d6-91d3-28f198c77d88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66\" (UID: \"fe530dfb-4534-49d6-91d3-28f198c77d88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"
Oct 30 21:15:00.524752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.524688    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91a2f39-f9cc-4bd9-8794-98d575ca8481-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q\" (UID: \"f91a2f39-f9cc-4bd9-8794-98d575ca8481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"
Oct 30 21:15:00.524752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.524726    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcn9w\" (UniqueName: \"kubernetes.io/projected/84174e23-a505-4963-a57e-41762b1faec7-kube-api-access-lcn9w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr\" (UID: \"84174e23-a505-4963-a57e-41762b1faec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"
Oct 30 21:15:00.524934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.524798    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3328918f-c5a6-4562-acbe-7d0f10ec1329-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-6wv5s\" (UID: \"3328918f-c5a6-4562-acbe-7d0f10ec1329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"
Oct 30 21:15:00.524934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.524872    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69815735-5039-4348-ae5d-68881b8555bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg\" (UID: \"69815735-5039-4348-ae5d-68881b8555bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"
Oct 30 21:15:00.524934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.524927    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxt4\" (UniqueName: \"kubernetes.io/projected/774fc560-ba2d-4285-8142-be854b22ffc6-kube-api-access-swxt4\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9\" (UID: \"774fc560-ba2d-4285-8142-be854b22ffc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"
Oct 30 21:15:00.525105 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.524963    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84174e23-a505-4963-a57e-41762b1faec7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr\" (UID: \"84174e23-a505-4963-a57e-41762b1faec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"
Oct 30 21:15:00.525105 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525004    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlqb2\" (UniqueName: \"kubernetes.io/projected/e9df251a-ccad-45c9-a477-2fa0e8f67060-kube-api-access-jlqb2\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj\" (UID: \"e9df251a-ccad-45c9-a477-2fa0e8f67060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"
Oct 30 21:15:00.525105 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525043    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29ca7de-8f46-4550-9e3b-58e215775d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24\" (UID: \"c29ca7de-8f46-4550-9e3b-58e215775d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"
Oct 30 21:15:00.525105 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525073    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbsv2\" (UniqueName: \"kubernetes.io/projected/f91a2f39-f9cc-4bd9-8794-98d575ca8481-kube-api-access-sbsv2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q\" (UID: \"f91a2f39-f9cc-4bd9-8794-98d575ca8481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"
Oct 30 21:15:00.525297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525206    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8j2f\" (UniqueName: \"kubernetes.io/projected/80ae6ddf-42ce-46fb-bae7-9facca564fbf-kube-api-access-k8j2f\") pod \"maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r\" (UID: \"80ae6ddf-42ce-46fb-bae7-9facca564fbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"
Oct 30 21:15:00.525297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525276    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rdr\" (UniqueName: \"kubernetes.io/projected/69815735-5039-4348-ae5d-68881b8555bd-kube-api-access-95rdr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg\" (UID: \"69815735-5039-4348-ae5d-68881b8555bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"
Oct 30 21:15:00.525441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525313    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4d5\" (UniqueName: \"kubernetes.io/projected/fe530dfb-4534-49d6-91d3-28f198c77d88-kube-api-access-kt4d5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66\" (UID: \"fe530dfb-4534-49d6-91d3-28f198c77d88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"
Oct 30 21:15:00.525441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525419    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25pc\" (UniqueName: \"kubernetes.io/projected/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-kube-api-access-s25pc\") pod \"maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt\" (UID: \"676f8fee-509a-4b6f-8fe0-ceaaae8f709a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"
Oct 30 21:15:00.525544 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525471    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqnw\" (UniqueName: \"kubernetes.io/projected/3328918f-c5a6-4562-acbe-7d0f10ec1329-kube-api-access-9fqnw\") pod \"maas-api-key-cleanup-29889915-6wv5s\" (UID: \"3328918f-c5a6-4562-acbe-7d0f10ec1329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"
Oct 30 21:15:00.525544 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525504    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ae6ddf-42ce-46fb-bae7-9facca564fbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r\" (UID: \"80ae6ddf-42ce-46fb-bae7-9facca564fbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"
Oct 30 21:15:00.525639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525541    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt\" (UID: \"676f8fee-509a-4b6f-8fe0-ceaaae8f709a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"
Oct 30 21:15:00.525639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525616    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/774fc560-ba2d-4285-8142-be854b22ffc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9\" (UID: \"774fc560-ba2d-4285-8142-be854b22ffc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"
Oct 30 21:15:00.525741 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.525653    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9df251a-ccad-45c9-a477-2fa0e8f67060-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj\" (UID: \"e9df251a-ccad-45c9-a477-2fa0e8f67060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"
Oct 30 21:15:00.532160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.527353    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29ca7de-8f46-4550-9e3b-58e215775d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24\" (UID: \"c29ca7de-8f46-4550-9e3b-58e215775d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"
Oct 30 21:15:00.532160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.528119    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91a2f39-f9cc-4bd9-8794-98d575ca8481-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q\" (UID: \"f91a2f39-f9cc-4bd9-8794-98d575ca8481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"
Oct 30 21:15:00.532160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.528738    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe530dfb-4534-49d6-91d3-28f198c77d88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66\" (UID: \"fe530dfb-4534-49d6-91d3-28f198c77d88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"
Oct 30 21:15:00.536486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.536461    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbsv2\" (UniqueName: \"kubernetes.io/projected/f91a2f39-f9cc-4bd9-8794-98d575ca8481-kube-api-access-sbsv2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q\" (UID: \"f91a2f39-f9cc-4bd9-8794-98d575ca8481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"
Oct 30 21:15:00.537295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.537267    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqc6c\" (UniqueName: \"kubernetes.io/projected/c29ca7de-8f46-4550-9e3b-58e215775d2d-kube-api-access-zqc6c\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24\" (UID: \"c29ca7de-8f46-4550-9e3b-58e215775d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"
Oct 30 21:15:00.538222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.538195    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4d5\" (UniqueName: \"kubernetes.io/projected/fe530dfb-4534-49d6-91d3-28f198c77d88-kube-api-access-kt4d5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66\" (UID: \"fe530dfb-4534-49d6-91d3-28f198c77d88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"
Oct 30 21:15:00.547160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.547140    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"
Oct 30 21:15:00.566249 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.566227    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"
Oct 30 21:15:00.617552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.617321    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"
Oct 30 21:15:00.628127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.628073    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt\" (UID: \"676f8fee-509a-4b6f-8fe0-ceaaae8f709a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"
Oct 30 21:15:00.629210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.629187    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt\" (UID: \"676f8fee-509a-4b6f-8fe0-ceaaae8f709a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"
Oct 30 21:15:00.629766 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.629743    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/774fc560-ba2d-4285-8142-be854b22ffc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9\" (UID: \"774fc560-ba2d-4285-8142-be854b22ffc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"
Oct 30 21:15:00.630898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.630810    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/774fc560-ba2d-4285-8142-be854b22ffc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9\" (UID: \"774fc560-ba2d-4285-8142-be854b22ffc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"
Oct 30 21:15:00.631098 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.631078    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9df251a-ccad-45c9-a477-2fa0e8f67060-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj\" (UID: \"e9df251a-ccad-45c9-a477-2fa0e8f67060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"
Oct 30 21:15:00.632180 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.632159    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcn9w\" (UniqueName: \"kubernetes.io/projected/84174e23-a505-4963-a57e-41762b1faec7-kube-api-access-lcn9w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr\" (UID: \"84174e23-a505-4963-a57e-41762b1faec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"
Oct 30 21:15:00.635499 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.635476    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3328918f-c5a6-4562-acbe-7d0f10ec1329-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-6wv5s\" (UID: \"3328918f-c5a6-4562-acbe-7d0f10ec1329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"
Oct 30 21:15:00.636461 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.636441    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69815735-5039-4348-ae5d-68881b8555bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg\" (UID: \"69815735-5039-4348-ae5d-68881b8555bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"
Oct 30 21:15:00.637485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.637465    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxt4\" (UniqueName: \"kubernetes.io/projected/774fc560-ba2d-4285-8142-be854b22ffc6-kube-api-access-swxt4\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9\" (UID: \"774fc560-ba2d-4285-8142-be854b22ffc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"
Oct 30 21:15:00.638708 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.637918    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84174e23-a505-4963-a57e-41762b1faec7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr\" (UID: \"84174e23-a505-4963-a57e-41762b1faec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"
Oct 30 21:15:00.638909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.636375    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3328918f-c5a6-4562-acbe-7d0f10ec1329-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-6wv5s\" (UID: \"3328918f-c5a6-4562-acbe-7d0f10ec1329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"
Oct 30 21:15:00.639346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.632045    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9df251a-ccad-45c9-a477-2fa0e8f67060-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj\" (UID: \"e9df251a-ccad-45c9-a477-2fa0e8f67060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"
Oct 30 21:15:00.639653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.638633    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84174e23-a505-4963-a57e-41762b1faec7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr\" (UID: \"84174e23-a505-4963-a57e-41762b1faec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"
Oct 30 21:15:00.639787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.637229    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69815735-5039-4348-ae5d-68881b8555bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg\" (UID: \"69815735-5039-4348-ae5d-68881b8555bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"
Oct 30 21:15:00.639940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.639921    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlqb2\" (UniqueName: \"kubernetes.io/projected/e9df251a-ccad-45c9-a477-2fa0e8f67060-kube-api-access-jlqb2\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj\" (UID: \"e9df251a-ccad-45c9-a477-2fa0e8f67060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"
Oct 30 21:15:00.640207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.640189    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8j2f\" (UniqueName: \"kubernetes.io/projected/80ae6ddf-42ce-46fb-bae7-9facca564fbf-kube-api-access-k8j2f\") pod \"maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r\" (UID: \"80ae6ddf-42ce-46fb-bae7-9facca564fbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"
Oct 30 21:15:00.640622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.640603    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rdr\" (UniqueName: \"kubernetes.io/projected/69815735-5039-4348-ae5d-68881b8555bd-kube-api-access-95rdr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg\" (UID: \"69815735-5039-4348-ae5d-68881b8555bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"
Oct 30 21:15:00.641085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.641065    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25pc\" (UniqueName: \"kubernetes.io/projected/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-kube-api-access-s25pc\") pod \"maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt\" (UID: \"676f8fee-509a-4b6f-8fe0-ceaaae8f709a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"
Oct 30 21:15:00.641487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.641465    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqnw\" (UniqueName: \"kubernetes.io/projected/3328918f-c5a6-4562-acbe-7d0f10ec1329-kube-api-access-9fqnw\") pod \"maas-api-key-cleanup-29889915-6wv5s\" (UID: \"3328918f-c5a6-4562-acbe-7d0f10ec1329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"
Oct 30 21:15:00.641850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.641831    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ae6ddf-42ce-46fb-bae7-9facca564fbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r\" (UID: \"80ae6ddf-42ce-46fb-bae7-9facca564fbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"
Oct 30 21:15:00.642746 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.642630    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ae6ddf-42ce-46fb-bae7-9facca564fbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r\" (UID: \"80ae6ddf-42ce-46fb-bae7-9facca564fbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"
Oct 30 21:15:00.649818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.649035    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcn9w\" (UniqueName: \"kubernetes.io/projected/84174e23-a505-4963-a57e-41762b1faec7-kube-api-access-lcn9w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr\" (UID: \"84174e23-a505-4963-a57e-41762b1faec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"
Oct 30 21:15:00.651548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.651519    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8j2f\" (UniqueName: \"kubernetes.io/projected/80ae6ddf-42ce-46fb-bae7-9facca564fbf-kube-api-access-k8j2f\") pod \"maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r\" (UID: \"80ae6ddf-42ce-46fb-bae7-9facca564fbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"
Oct 30 21:15:00.653370 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.653293    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqnw\" (UniqueName: \"kubernetes.io/projected/3328918f-c5a6-4562-acbe-7d0f10ec1329-kube-api-access-9fqnw\") pod \"maas-api-key-cleanup-29889915-6wv5s\" (UID: \"3328918f-c5a6-4562-acbe-7d0f10ec1329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"
Oct 30 21:15:00.653535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.653508    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25pc\" (UniqueName: \"kubernetes.io/projected/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-kube-api-access-s25pc\") pod \"maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt\" (UID: \"676f8fee-509a-4b6f-8fe0-ceaaae8f709a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"
Oct 30 21:15:00.654957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.654887    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxt4\" (UniqueName: \"kubernetes.io/projected/774fc560-ba2d-4285-8142-be854b22ffc6-kube-api-access-swxt4\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9\" (UID: \"774fc560-ba2d-4285-8142-be854b22ffc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"
Oct 30 21:15:00.656675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.656617    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rdr\" (UniqueName: \"kubernetes.io/projected/69815735-5039-4348-ae5d-68881b8555bd-kube-api-access-95rdr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg\" (UID: \"69815735-5039-4348-ae5d-68881b8555bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"
Oct 30 21:15:00.656675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.656640    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlqb2\" (UniqueName: \"kubernetes.io/projected/e9df251a-ccad-45c9-a477-2fa0e8f67060-kube-api-access-jlqb2\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj\" (UID: \"e9df251a-ccad-45c9-a477-2fa0e8f67060\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"
Oct 30 21:15:00.678264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.678232    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"
Oct 30 21:15:00.720599 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.720352    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"
Oct 30 21:15:00.726615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.724516    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"
Oct 30 21:15:00.800933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.785692    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"
Oct 30 21:15:00.807057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.801622    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"
Oct 30 21:15:00.807057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.801806    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"
Oct 30 21:15:00.807057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.801926    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"
Oct 30 21:15:00.807057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.806476    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"
Oct 30 21:15:00.807057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.806582    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"
Oct 30 21:15:00.840137 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.839148    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"
Oct 30 21:15:00.842696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.842669    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"
Oct 30 21:15:00.849479 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.849345    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"
Oct 30 21:15:00.963663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.963035    4810 generic.go:356] "Generic (PLEG): container finished" podID="a5b886ae-5b1c-43fb-94ff-92c7ac1f3683" containerID="ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621" exitCode=0
Oct 30 21:15:00.963663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.963221    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97" event={"ID":"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683","Type":"ContainerDied","Data":"ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621"}
Oct 30 21:15:00.963663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.963279    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97" event={"ID":"a5b886ae-5b1c-43fb-94ff-92c7ac1f3683","Type":"ContainerDied","Data":"4d59b5f33902a460f04e49c13d80e897eacade538ade1d76758f6062a0274aa5"}
Oct 30 21:15:00.963663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.963312    4810 scope.go:122] "RemoveContainer" containerID="ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621"
Oct 30 21:15:00.963663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:00.963463    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"
Oct 30 21:15:01.083115 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:01.074628    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e47096_ee02_476d_ab4c_d7d0846a976a.slice/crio-1e4e338dcb970426a091797b585c46a080d9dbf0849b593fbc0d4bb624391757 WatchSource:0}: Error finding container 1e4e338dcb970426a091797b585c46a080d9dbf0849b593fbc0d4bb624391757: Status 404 returned error can't find the container with id 1e4e338dcb970426a091797b585c46a080d9dbf0849b593fbc0d4bb624391757
Oct 30 21:15:01.083115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.081264    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"]
Oct 30 21:15:01.095426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.095402    4810 scope.go:122] "RemoveContainer" containerID="ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621"
Oct 30 21:15:01.106261 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:01.100052    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621\": container with ID starting with ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621 not found: ID does not exist" containerID="ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621"
Oct 30 21:15:01.106261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.100098    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621"} err="failed to get container status \"ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621\": rpc error: code = NotFound desc = could not find container \"ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621\": container with ID starting with ab9dff837a89f3db1c66cd4ff53b55258086fd7989f38dd764910ec1a68e2621 not found: ID does not exist"
Oct 30 21:15:01.106261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.105296    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"]
Oct 30 21:15:01.115883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.115291    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bfhbn97"]
Oct 30 21:15:01.325794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.325765    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"]
Oct 30 21:15:01.345481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.345454    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"]
Oct 30 21:15:01.478614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.478585    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"]
Oct 30 21:15:01.525069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.525036    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"]
Oct 30 21:15:01.557218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.557188    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"]
Oct 30 21:15:01.564607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.564580    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"]
Oct 30 21:15:01.580616 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:01.580582    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18183221_a049_4171_add0_1810e4939630.slice/crio-16dc02e520257c65fdcf76c67276e15c413ac3e53676c4a7682336962132f76c WatchSource:0}: Error finding container 16dc02e520257c65fdcf76c67276e15c413ac3e53676c4a7682336962132f76c: Status 404 returned error can't find the container with id 16dc02e520257c65fdcf76c67276e15c413ac3e53676c4a7682336962132f76c
Oct 30 21:15:01.583922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.583894    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"]
Oct 30 21:15:01.588894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.588842    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"]
Oct 30 21:15:01.608627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.608590    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"]
Oct 30 21:15:01.729133 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:01.728988    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9df251a_ccad_45c9_a477_2fa0e8f67060.slice/crio-c1f9904c6eab3c3dfc500914f56bb59cd3fe4833c0c5bce99265159987ae8a44 WatchSource:0}: Error finding container c1f9904c6eab3c3dfc500914f56bb59cd3fe4833c0c5bce99265159987ae8a44: Status 404 returned error can't find the container with id c1f9904c6eab3c3dfc500914f56bb59cd3fe4833c0c5bce99265159987ae8a44
Oct 30 21:15:01.735635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.735605    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b886ae-5b1c-43fb-94ff-92c7ac1f3683" path="/var/lib/kubelet/pods/a5b886ae-5b1c-43fb-94ff-92c7ac1f3683/volumes"
Oct 30 21:15:01.736228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.736204    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"]
Oct 30 21:15:01.736333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.736272    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"]
Oct 30 21:15:01.736333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.736314    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"]
Oct 30 21:15:01.736440 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.736351    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"]
Oct 30 21:15:01.747564 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.747537    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"]
Oct 30 21:15:01.759237 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:01.759212    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe530dfb_4534_49d6_91d3_28f198c77d88.slice/crio-4357107acfe1dfbf1265b043bc24188165aa77e12da8bfec3776945d29bd1df4 WatchSource:0}: Error finding container 4357107acfe1dfbf1265b043bc24188165aa77e12da8bfec3776945d29bd1df4: Status 404 returned error can't find the container with id 4357107acfe1dfbf1265b043bc24188165aa77e12da8bfec3776945d29bd1df4
Oct 30 21:15:01.973569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.973471    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" event={"ID":"84174e23-a505-4963-a57e-41762b1faec7","Type":"ContainerStarted","Data":"e0a34bc047ef3c4896a1624af77bdb79197b73b618a93acf5fe54a22e201d7cb"}
Oct 30 21:15:01.985894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.976750    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm" event={"ID":"fb4d6041-3a3c-4980-9f2f-849b4bb7d457","Type":"ContainerStarted","Data":"983e2dda42d2ae6f8d1595ec12ccd29e992fcf25008ab219068bc2ad437af51f"}
Oct 30 21:15:01.985894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.982251    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" event={"ID":"774fc560-ba2d-4285-8142-be854b22ffc6","Type":"ContainerStarted","Data":"0c03591a58c7fe1105fb6fe2b943a1aa1c0f0cb540c6a90c9948fab4c26d4d86"}
Oct 30 21:15:01.988104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.988073    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" event={"ID":"80ae6ddf-42ce-46fb-bae7-9facca564fbf","Type":"ContainerStarted","Data":"16f505763732c629781cc949afd96fa85f3c9d4e85044c1a5916534ba97c9276"}
Oct 30 21:15:01.993825 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.993793    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" event={"ID":"e9df251a-ccad-45c9-a477-2fa0e8f67060","Type":"ContainerStarted","Data":"c1f9904c6eab3c3dfc500914f56bb59cd3fe4833c0c5bce99265159987ae8a44"}
Oct 30 21:15:01.995302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.995277    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" event={"ID":"3328918f-c5a6-4562-acbe-7d0f10ec1329","Type":"ContainerStarted","Data":"42ff05e06c3f501f747738a0343a82f27ba3c23cf6215d7a4cc9100894e15e35"}
Oct 30 21:15:01.997094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.996842    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" event={"ID":"fe530dfb-4534-49d6-91d3-28f198c77d88","Type":"ContainerStarted","Data":"4357107acfe1dfbf1265b043bc24188165aa77e12da8bfec3776945d29bd1df4"}
Oct 30 21:15:01.998139 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.998100    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629" event={"ID":"18183221-a049-4171-add0-1810e4939630","Type":"ContainerStarted","Data":"16dc02e520257c65fdcf76c67276e15c413ac3e53676c4a7682336962132f76c"}
Oct 30 21:15:02.000225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.999908    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" event={"ID":"a0e47096-ee02-476d-ab4c-d7d0846a976a","Type":"ContainerStarted","Data":"3c643f4eab93d288e5fab74162e0b084938d1b0ced66d71cf71d562fb9d113e9"}
Oct 30 21:15:02.000225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:01.999953    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" event={"ID":"a0e47096-ee02-476d-ab4c-d7d0846a976a","Type":"ContainerStarted","Data":"1e4e338dcb970426a091797b585c46a080d9dbf0849b593fbc0d4bb624391757"}
Oct 30 21:15:02.001580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:02.001552    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" event={"ID":"f91a2f39-f9cc-4bd9-8794-98d575ca8481","Type":"ContainerStarted","Data":"559c8da1ee331ef814e178a8265471a5af7794d379639167a91357a08bf33368"}
Oct 30 21:15:02.004369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:02.004328    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg" event={"ID":"69815735-5039-4348-ae5d-68881b8555bd","Type":"ContainerStarted","Data":"aa28115569b81e205fc9a8fd271f3fdd0b6582dc5c0d02aab8fa09608ed5c54c"}
Oct 30 21:15:02.005673 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:02.005614    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" event={"ID":"676f8fee-509a-4b6f-8fe0-ceaaae8f709a","Type":"ContainerStarted","Data":"48182dfa7a0e01221774790476bd7793b51e886c1f87fd110e1cc7cfa4f3671c"}
Oct 30 21:15:02.007205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:02.006940    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g" event={"ID":"2d1184f0-d306-4de0-a4f3-016c48e5f897","Type":"ContainerStarted","Data":"f6d137d1b838749db4ee25f85f1c5fb9cae33a185b0f1b88fa438c810396d1cf"}
Oct 30 21:15:02.008971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:02.008686    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts" event={"ID":"def80bf3-e036-460e-9543-114a136c81fe","Type":"ContainerStarted","Data":"1ffcf4a45b9016e14f4c8c88a88b488e565f024f10043202f27229c30b1d42b3"}
Oct 30 21:15:02.011516 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:02.011474    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" event={"ID":"c29ca7de-8f46-4550-9e3b-58e215775d2d","Type":"ContainerStarted","Data":"260a75dec603afe53c312a32f920af1b24f2a7b810272cab84e85f82adb04205"}
Oct 30 21:15:02.016325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:02.016280    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" podStartSLOduration=2.016263107 podStartE2EDuration="2.016263107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.015971138 +0000 UTC m=+2268.507787682" watchObservedRunningTime="2026-10-30 21:15:02.016263107 +0000 UTC m=+2268.508079655"
Oct 30 21:15:03.026114 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.026080    4810 generic.go:356] "Generic (PLEG): container finished" podID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerID="7768e4765a85058979d207dd1b855e560ecf11969d23ca3def984dd25a0c9f61" exitCode=60
Oct 30 21:15:03.026542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.026207    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm" event={"ID":"fb4d6041-3a3c-4980-9f2f-849b4bb7d457","Type":"ContainerDied","Data":"7768e4765a85058979d207dd1b855e560ecf11969d23ca3def984dd25a0c9f61"}
Oct 30 21:15:03.026542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.026501    4810 scope.go:122] "RemoveContainer" containerID="7768e4765a85058979d207dd1b855e560ecf11969d23ca3def984dd25a0c9f61"
Oct 30 21:15:03.029024 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.028994    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" event={"ID":"774fc560-ba2d-4285-8142-be854b22ffc6","Type":"ContainerStarted","Data":"3f55ccc1ea3d658004075b0490aba41ad2d8c2662ce5c00348afe802d2b91cc3"}
Oct 30 21:15:03.030957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.030909    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" event={"ID":"80ae6ddf-42ce-46fb-bae7-9facca564fbf","Type":"ContainerStarted","Data":"9d11d5a27f5b9d6f166cd1eac5a19b2dc953bb53c715b1c4fccad25af4807c3d"}
Oct 30 21:15:03.032655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.032632    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" event={"ID":"3328918f-c5a6-4562-acbe-7d0f10ec1329","Type":"ContainerStarted","Data":"15314da0385532fe3aaae419491e7d56470b842089842f1a7fd546996c9d659f"}
Oct 30 21:15:03.034548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.034529    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" event={"ID":"fe530dfb-4534-49d6-91d3-28f198c77d88","Type":"ContainerStarted","Data":"23d3daff315934babaeb8e241ccb4daca8a02fb009d5f49bcbbffa6415cc9f1a"}
Oct 30 21:15:03.036185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.036165    4810 generic.go:356] "Generic (PLEG): container finished" podID="18183221-a049-4171-add0-1810e4939630" containerID="6d8bc0f3dc264cfaf180fb7baa536f61644599b3fc230fa51db04f4ce897b1cc" exitCode=60
Oct 30 21:15:03.036278 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.036242    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629" event={"ID":"18183221-a049-4171-add0-1810e4939630","Type":"ContainerDied","Data":"6d8bc0f3dc264cfaf180fb7baa536f61644599b3fc230fa51db04f4ce897b1cc"}
Oct 30 21:15:03.036394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.036379    4810 scope.go:122] "RemoveContainer" containerID="6d8bc0f3dc264cfaf180fb7baa536f61644599b3fc230fa51db04f4ce897b1cc"
Oct 30 21:15:03.037980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.037960    4810 generic.go:356] "Generic (PLEG): container finished" podID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerID="3c643f4eab93d288e5fab74162e0b084938d1b0ced66d71cf71d562fb9d113e9" exitCode=60
Oct 30 21:15:03.038054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.038035    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" event={"ID":"a0e47096-ee02-476d-ab4c-d7d0846a976a","Type":"ContainerDied","Data":"3c643f4eab93d288e5fab74162e0b084938d1b0ced66d71cf71d562fb9d113e9"}
Oct 30 21:15:03.038221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.038205    4810 scope.go:122] "RemoveContainer" containerID="3c643f4eab93d288e5fab74162e0b084938d1b0ced66d71cf71d562fb9d113e9"
Oct 30 21:15:03.039705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.039684    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" event={"ID":"f91a2f39-f9cc-4bd9-8794-98d575ca8481","Type":"ContainerStarted","Data":"5e24f80f25dd5df8e4283546607ed9eb106853dc5ab928fdd875fffb267c7716"}
Oct 30 21:15:03.043362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.043307    4810 generic.go:356] "Generic (PLEG): container finished" podID="69815735-5039-4348-ae5d-68881b8555bd" containerID="09c47c2e47f80106fec54d9c656f4b688f1f010e556667a5ca131e81bf1e6f9a" exitCode=60
Oct 30 21:15:03.043495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.043480    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg" event={"ID":"69815735-5039-4348-ae5d-68881b8555bd","Type":"ContainerDied","Data":"09c47c2e47f80106fec54d9c656f4b688f1f010e556667a5ca131e81bf1e6f9a"}
Oct 30 21:15:03.043644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.043621    4810 scope.go:122] "RemoveContainer" containerID="09c47c2e47f80106fec54d9c656f4b688f1f010e556667a5ca131e81bf1e6f9a"
Oct 30 21:15:03.050317 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.050288    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" event={"ID":"676f8fee-509a-4b6f-8fe0-ceaaae8f709a","Type":"ContainerStarted","Data":"c0645065290f70ccfe605f8816552eb6de551316e1bb9b927e291523b16d9ef1"}
Oct 30 21:15:03.052164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.052144    4810 generic.go:356] "Generic (PLEG): container finished" podID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerID="3135510ab39d5829610525a6aa45836c7129cbc53ceeb923e6c7ff698b121499" exitCode=60
Oct 30 21:15:03.052256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.052228    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g" event={"ID":"2d1184f0-d306-4de0-a4f3-016c48e5f897","Type":"ContainerDied","Data":"3135510ab39d5829610525a6aa45836c7129cbc53ceeb923e6c7ff698b121499"}
Oct 30 21:15:03.052478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.052460    4810 scope.go:122] "RemoveContainer" containerID="3135510ab39d5829610525a6aa45836c7129cbc53ceeb923e6c7ff698b121499"
Oct 30 21:15:03.055040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.054273    4810 generic.go:356] "Generic (PLEG): container finished" podID="def80bf3-e036-460e-9543-114a136c81fe" containerID="655ec16bd7f8ecce25360b4c475eef9ea4f12ecc56d3e142aa111849b4fec4cc" exitCode=60
Oct 30 21:15:03.055040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.054347    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts" event={"ID":"def80bf3-e036-460e-9543-114a136c81fe","Type":"ContainerDied","Data":"655ec16bd7f8ecce25360b4c475eef9ea4f12ecc56d3e142aa111849b4fec4cc"}
Oct 30 21:15:03.055040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.054626    4810 scope.go:122] "RemoveContainer" containerID="655ec16bd7f8ecce25360b4c475eef9ea4f12ecc56d3e142aa111849b4fec4cc"
Oct 30 21:15:03.056391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.056368    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" event={"ID":"c29ca7de-8f46-4550-9e3b-58e215775d2d","Type":"ContainerStarted","Data":"5dfa9fc7b1eaa75255b46fc44e60d40a8d296fd7cfd7c785c45b031477909bd7"}
Oct 30 21:15:03.058143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.058124    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" event={"ID":"84174e23-a505-4963-a57e-41762b1faec7","Type":"ContainerStarted","Data":"d94944382112dca94bd1a6e147bf8721d178b4fa7553f23d82ce0d538d5f20cf"}
Oct 30 21:15:03.078384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.073704    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" podStartSLOduration=3.073688791 podStartE2EDuration="3.073688791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.070504051 +0000 UTC m=+2269.562320606" watchObservedRunningTime="2026-10-30 21:15:03.073688791 +0000 UTC m=+2269.565505336"
Oct 30 21:15:03.095377 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.093934    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" podStartSLOduration=3.09391567 podStartE2EDuration="3.09391567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.092065719 +0000 UTC m=+2269.583882290" watchObservedRunningTime="2026-10-30 21:15:03.09391567 +0000 UTC m=+2269.585732406"
Oct 30 21:15:03.137872 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.137665    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" podStartSLOduration=3.137646669 podStartE2EDuration="3.137646669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.134202094 +0000 UTC m=+2269.626018640" watchObservedRunningTime="2026-10-30 21:15:03.137646669 +0000 UTC m=+2269.629463215"
Oct 30 21:15:03.169574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.169500    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" podStartSLOduration=3.16947268 podStartE2EDuration="3.16947268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.166689344 +0000 UTC m=+2269.658505879" watchObservedRunningTime="2026-10-30 21:15:03.16947268 +0000 UTC m=+2269.661289218"
Oct 30 21:15:03.220953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.220894    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" podStartSLOduration=3.220841155 podStartE2EDuration="3.220841155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.218885508 +0000 UTC m=+2269.710702053" watchObservedRunningTime="2026-10-30 21:15:03.220841155 +0000 UTC m=+2269.712657700"
Oct 30 21:15:03.264663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.264608    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" podStartSLOduration=3.264594622 podStartE2EDuration="3.264594622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.261112477 +0000 UTC m=+2269.752929021" watchObservedRunningTime="2026-10-30 21:15:03.264594622 +0000 UTC m=+2269.756411167"
Oct 30 21:15:03.299869 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.299800    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" podStartSLOduration=3.299785542 podStartE2EDuration="3.299785542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.295619283 +0000 UTC m=+2269.787435827" watchObservedRunningTime="2026-10-30 21:15:03.299785542 +0000 UTC m=+2269.791602087"
Oct 30 21:15:03.314406 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:03.314083    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" podStartSLOduration=3.314069362 podStartE2EDuration="3.314069362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.312252394 +0000 UTC m=+2269.804068941" watchObservedRunningTime="2026-10-30 21:15:03.314069362 +0000 UTC m=+2269.805885906"
Oct 30 21:15:04.068795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.068757    4810 generic.go:356] "Generic (PLEG): container finished" podID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerID="c0645065290f70ccfe605f8816552eb6de551316e1bb9b927e291523b16d9ef1" exitCode=60
Oct 30 21:15:04.069264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.068843    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" event={"ID":"676f8fee-509a-4b6f-8fe0-ceaaae8f709a","Type":"ContainerDied","Data":"c0645065290f70ccfe605f8816552eb6de551316e1bb9b927e291523b16d9ef1"}
Oct 30 21:15:04.069264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.069182    4810 scope.go:122] "RemoveContainer" containerID="c0645065290f70ccfe605f8816552eb6de551316e1bb9b927e291523b16d9ef1"
Oct 30 21:15:04.070687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.070667    4810 generic.go:356] "Generic (PLEG): container finished" podID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerID="5dfa9fc7b1eaa75255b46fc44e60d40a8d296fd7cfd7c785c45b031477909bd7" exitCode=60
Oct 30 21:15:04.070775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.070748    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" event={"ID":"c29ca7de-8f46-4550-9e3b-58e215775d2d","Type":"ContainerDied","Data":"5dfa9fc7b1eaa75255b46fc44e60d40a8d296fd7cfd7c785c45b031477909bd7"}
Oct 30 21:15:04.071032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.071013    4810 scope.go:122] "RemoveContainer" containerID="5dfa9fc7b1eaa75255b46fc44e60d40a8d296fd7cfd7c785c45b031477909bd7"
Oct 30 21:15:04.084486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.082062    4810 generic.go:356] "Generic (PLEG): container finished" podID="84174e23-a505-4963-a57e-41762b1faec7" containerID="d94944382112dca94bd1a6e147bf8721d178b4fa7553f23d82ce0d538d5f20cf" exitCode=60
Oct 30 21:15:04.084486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.082205    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" event={"ID":"84174e23-a505-4963-a57e-41762b1faec7","Type":"ContainerDied","Data":"d94944382112dca94bd1a6e147bf8721d178b4fa7553f23d82ce0d538d5f20cf"}
Oct 30 21:15:04.084486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.082604    4810 scope.go:122] "RemoveContainer" containerID="d94944382112dca94bd1a6e147bf8721d178b4fa7553f23d82ce0d538d5f20cf"
Oct 30 21:15:04.086400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.086372    4810 generic.go:356] "Generic (PLEG): container finished" podID="774fc560-ba2d-4285-8142-be854b22ffc6" containerID="3f55ccc1ea3d658004075b0490aba41ad2d8c2662ce5c00348afe802d2b91cc3" exitCode=60
Oct 30 21:15:04.086513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.086460    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" event={"ID":"774fc560-ba2d-4285-8142-be854b22ffc6","Type":"ContainerDied","Data":"3f55ccc1ea3d658004075b0490aba41ad2d8c2662ce5c00348afe802d2b91cc3"}
Oct 30 21:15:04.086709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.086691    4810 scope.go:122] "RemoveContainer" containerID="3f55ccc1ea3d658004075b0490aba41ad2d8c2662ce5c00348afe802d2b91cc3"
Oct 30 21:15:04.095233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.089818    4810 generic.go:356] "Generic (PLEG): container finished" podID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerID="9d11d5a27f5b9d6f166cd1eac5a19b2dc953bb53c715b1c4fccad25af4807c3d" exitCode=60
Oct 30 21:15:04.095233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.089937    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" event={"ID":"80ae6ddf-42ce-46fb-bae7-9facca564fbf","Type":"ContainerDied","Data":"9d11d5a27f5b9d6f166cd1eac5a19b2dc953bb53c715b1c4fccad25af4807c3d"}
Oct 30 21:15:04.095233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.092099    4810 generic.go:356] "Generic (PLEG): container finished" podID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerID="2cff708ec19e14a3bd10f4d36b5091711f0b15a817ee43a7bd4dbcfaf368db73" exitCode=60
Oct 30 21:15:04.095233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.092273    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" event={"ID":"e9df251a-ccad-45c9-a477-2fa0e8f67060","Type":"ContainerDied","Data":"2cff708ec19e14a3bd10f4d36b5091711f0b15a817ee43a7bd4dbcfaf368db73"}
Oct 30 21:15:04.095233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.092611    4810 scope.go:122] "RemoveContainer" containerID="2cff708ec19e14a3bd10f4d36b5091711f0b15a817ee43a7bd4dbcfaf368db73"
Oct 30 21:15:04.097087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.096846    4810 scope.go:122] "RemoveContainer" containerID="9d11d5a27f5b9d6f166cd1eac5a19b2dc953bb53c715b1c4fccad25af4807c3d"
Oct 30 21:15:04.110449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.109353    4810 generic.go:356] "Generic (PLEG): container finished" podID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerID="15314da0385532fe3aaae419491e7d56470b842089842f1a7fd546996c9d659f" exitCode=60
Oct 30 21:15:04.110449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.109452    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" event={"ID":"3328918f-c5a6-4562-acbe-7d0f10ec1329","Type":"ContainerDied","Data":"15314da0385532fe3aaae419491e7d56470b842089842f1a7fd546996c9d659f"}
Oct 30 21:15:04.110449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.109976    4810 scope.go:122] "RemoveContainer" containerID="15314da0385532fe3aaae419491e7d56470b842089842f1a7fd546996c9d659f"
Oct 30 21:15:04.116403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.116374    4810 generic.go:356] "Generic (PLEG): container finished" podID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerID="23d3daff315934babaeb8e241ccb4daca8a02fb009d5f49bcbbffa6415cc9f1a" exitCode=60
Oct 30 21:15:04.116513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.116460    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" event={"ID":"fe530dfb-4534-49d6-91d3-28f198c77d88","Type":"ContainerDied","Data":"23d3daff315934babaeb8e241ccb4daca8a02fb009d5f49bcbbffa6415cc9f1a"}
Oct 30 21:15:04.116754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.116735    4810 scope.go:122] "RemoveContainer" containerID="23d3daff315934babaeb8e241ccb4daca8a02fb009d5f49bcbbffa6415cc9f1a"
Oct 30 21:15:04.121047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.120782    4810 generic.go:356] "Generic (PLEG): container finished" podID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerID="5e24f80f25dd5df8e4283546607ed9eb106853dc5ab928fdd875fffb267c7716" exitCode=60
Oct 30 21:15:04.121047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.120823    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" event={"ID":"f91a2f39-f9cc-4bd9-8794-98d575ca8481","Type":"ContainerDied","Data":"5e24f80f25dd5df8e4283546607ed9eb106853dc5ab928fdd875fffb267c7716"}
Oct 30 21:15:04.121294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:04.121275    4810 scope.go:122] "RemoveContainer" containerID="5e24f80f25dd5df8e4283546607ed9eb106853dc5ab928fdd875fffb267c7716"
Oct 30 21:15:05.133910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.133872    4810 generic.go:356] "Generic (PLEG): container finished" podID="69815735-5039-4348-ae5d-68881b8555bd" containerID="6d1446f2726bc242398648a9e6c53e4a56a0f3ce0774c816da79bf6945e80546" exitCode=60
Oct 30 21:15:05.134324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.133884    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg" event={"ID":"69815735-5039-4348-ae5d-68881b8555bd","Type":"ContainerDied","Data":"6d1446f2726bc242398648a9e6c53e4a56a0f3ce0774c816da79bf6945e80546"}
Oct 30 21:15:05.134324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.134043    4810 scope.go:122] "RemoveContainer" containerID="09c47c2e47f80106fec54d9c656f4b688f1f010e556667a5ca131e81bf1e6f9a"
Oct 30 21:15:05.134425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.134346    4810 scope.go:122] "RemoveContainer" containerID="6d1446f2726bc242398648a9e6c53e4a56a0f3ce0774c816da79bf6945e80546"
Oct 30 21:15:05.134824 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:05.134766    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg_odh-ai-gateway-infra(69815735-5039-4348-ae5d-68881b8555bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg" podUID="69815735-5039-4348-ae5d-68881b8555bd"
Oct 30 21:15:05.136895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.136869    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" event={"ID":"676f8fee-509a-4b6f-8fe0-ceaaae8f709a","Type":"ContainerStarted","Data":"bef57a34aefd550a45ea1912771ce800ebdb71a0b4b7ab5d16f2b6fe1b5ab7d3"}
Oct 30 21:15:05.139552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.139527    4810 generic.go:356] "Generic (PLEG): container finished" podID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerID="ca89de7a4227267e5347099eaa0ba9ece425f6be15a762595d8d51a02d8e2e3d" exitCode=60
Oct 30 21:15:05.139692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.139610    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g" event={"ID":"2d1184f0-d306-4de0-a4f3-016c48e5f897","Type":"ContainerDied","Data":"ca89de7a4227267e5347099eaa0ba9ece425f6be15a762595d8d51a02d8e2e3d"}
Oct 30 21:15:05.140017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.139993    4810 scope.go:122] "RemoveContainer" containerID="ca89de7a4227267e5347099eaa0ba9ece425f6be15a762595d8d51a02d8e2e3d"
Oct 30 21:15:05.140399 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:05.140375    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g_odh-ai-gateway-infra(2d1184f0-d306-4de0-a4f3-016c48e5f897)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897"
Oct 30 21:15:05.142489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.142389    4810 generic.go:356] "Generic (PLEG): container finished" podID="def80bf3-e036-460e-9543-114a136c81fe" containerID="72fe7ba4d4a56a843fffa0e72bc0cd329c5d21d25237f7c8de618382b8efc893" exitCode=60
Oct 30 21:15:05.142489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.142416    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts" event={"ID":"def80bf3-e036-460e-9543-114a136c81fe","Type":"ContainerDied","Data":"72fe7ba4d4a56a843fffa0e72bc0cd329c5d21d25237f7c8de618382b8efc893"}
Oct 30 21:15:05.143606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.143282    4810 scope.go:122] "RemoveContainer" containerID="72fe7ba4d4a56a843fffa0e72bc0cd329c5d21d25237f7c8de618382b8efc893"
Oct 30 21:15:05.143606 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:05.143561    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts_odh-ai-gateway-infra(def80bf3-e036-460e-9543-114a136c81fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts" podUID="def80bf3-e036-460e-9543-114a136c81fe"
Oct 30 21:15:05.144928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.144906    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" event={"ID":"c29ca7de-8f46-4550-9e3b-58e215775d2d","Type":"ContainerStarted","Data":"d833d7f74526bdb84d58d45063bab735a8fc861c7469b7cd29555de038ee88b8"}
Oct 30 21:15:05.147384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.147365    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" event={"ID":"84174e23-a505-4963-a57e-41762b1faec7","Type":"ContainerStarted","Data":"75b9b8c75f8bef25a355d5b56c3762626dc058f19d6c55aa58f137c7c9541de4"}
Oct 30 21:15:05.150232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.150210    4810 generic.go:356] "Generic (PLEG): container finished" podID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerID="5414ac07170ad8ea24410be54c8639512330b104d5b07331331ddc63a76dba02" exitCode=60
Oct 30 21:15:05.150327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.150281    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm" event={"ID":"fb4d6041-3a3c-4980-9f2f-849b4bb7d457","Type":"ContainerDied","Data":"5414ac07170ad8ea24410be54c8639512330b104d5b07331331ddc63a76dba02"}
Oct 30 21:15:05.150558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.150541    4810 scope.go:122] "RemoveContainer" containerID="5414ac07170ad8ea24410be54c8639512330b104d5b07331331ddc63a76dba02"
Oct 30 21:15:05.150833 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:05.150812    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm_odh-ai-gateway-infra(fb4d6041-3a3c-4980-9f2f-849b4bb7d457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457"
Oct 30 21:15:05.152508 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.152484    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" event={"ID":"774fc560-ba2d-4285-8142-be854b22ffc6","Type":"ContainerStarted","Data":"1bfb8df3a5bde0241ae1b869be225a5a39360ca0cc30dc9b44c7688384bbc26a"}
Oct 30 21:15:05.153519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.153498    4810 scope.go:122] "RemoveContainer" containerID="3135510ab39d5829610525a6aa45836c7129cbc53ceeb923e6c7ff698b121499"
Oct 30 21:15:05.164907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.164835    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" event={"ID":"80ae6ddf-42ce-46fb-bae7-9facca564fbf","Type":"ContainerStarted","Data":"4fbd63963785dacfc8cc987281c9809eacf09a631e7f8e5bc00ade17824062ed"}
Oct 30 21:15:05.167090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.167059    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" event={"ID":"e9df251a-ccad-45c9-a477-2fa0e8f67060","Type":"ContainerStarted","Data":"d921e532e3d34872443ff649a36b8960c169963a5516d315bbc12e05bc94a287"}
Oct 30 21:15:05.169389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.169364    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" event={"ID":"3328918f-c5a6-4562-acbe-7d0f10ec1329","Type":"ContainerStarted","Data":"2cc43f432f6bc6247d2cd904e296b5549f7f733b240e75bf358236f37e7a8c10"}
Oct 30 21:15:05.184538 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.183885    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" event={"ID":"fe530dfb-4534-49d6-91d3-28f198c77d88","Type":"ContainerStarted","Data":"e9c7b2bbde0b5738a7198c0efe5df2d5fa20a1e2138274bec869420f613d370d"}
Oct 30 21:15:05.188435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.187413    4810 generic.go:356] "Generic (PLEG): container finished" podID="18183221-a049-4171-add0-1810e4939630" containerID="6111b830195345e2370d9a1712847a4f30e59a264571b88643cd2125c888c300" exitCode=60
Oct 30 21:15:05.188435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.187520    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629" event={"ID":"18183221-a049-4171-add0-1810e4939630","Type":"ContainerDied","Data":"6111b830195345e2370d9a1712847a4f30e59a264571b88643cd2125c888c300"}
Oct 30 21:15:05.188435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.187972    4810 scope.go:122] "RemoveContainer" containerID="6111b830195345e2370d9a1712847a4f30e59a264571b88643cd2125c888c300"
Oct 30 21:15:05.188435 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:05.188254    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629_odh-ai-gateway-infra(18183221-a049-4171-add0-1810e4939630)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629" podUID="18183221-a049-4171-add0-1810e4939630"
Oct 30 21:15:05.194392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.194368    4810 generic.go:356] "Generic (PLEG): container finished" podID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerID="fa2c13c4ea2cc7185d6531036857e9424f9e60b0d7f4b04ccd7b3ff831b12d28" exitCode=60
Oct 30 21:15:05.194526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.194429    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" event={"ID":"a0e47096-ee02-476d-ab4c-d7d0846a976a","Type":"ContainerDied","Data":"fa2c13c4ea2cc7185d6531036857e9424f9e60b0d7f4b04ccd7b3ff831b12d28"}
Oct 30 21:15:05.195074 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.194710    4810 scope.go:122] "RemoveContainer" containerID="fa2c13c4ea2cc7185d6531036857e9424f9e60b0d7f4b04ccd7b3ff831b12d28"
Oct 30 21:15:05.195074 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:05.195022    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n_odh-ai-gateway-infra(a0e47096-ee02-476d-ab4c-d7d0846a976a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a"
Oct 30 21:15:05.199067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.199041    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" event={"ID":"f91a2f39-f9cc-4bd9-8794-98d575ca8481","Type":"ContainerStarted","Data":"a1119e0608305dab35170f691deaef9a6daa59e114cbf1f6f9cc01b35098f798"}
Oct 30 21:15:05.200347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.200327    4810 scope.go:122] "RemoveContainer" containerID="655ec16bd7f8ecce25360b4c475eef9ea4f12ecc56d3e142aa111849b4fec4cc"
Oct 30 21:15:05.217722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.217702    4810 scope.go:122] "RemoveContainer" containerID="7768e4765a85058979d207dd1b855e560ecf11969d23ca3def984dd25a0c9f61"
Oct 30 21:15:05.232783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.232760    4810 scope.go:122] "RemoveContainer" containerID="6d8bc0f3dc264cfaf180fb7baa536f61644599b3fc230fa51db04f4ce897b1cc"
Oct 30 21:15:05.247297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.247277    4810 scope.go:122] "RemoveContainer" containerID="3c643f4eab93d288e5fab74162e0b084938d1b0ced66d71cf71d562fb9d113e9"
Oct 30 21:15:05.419098 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:05.419052    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" podStartSLOduration=5.419038663 podStartE2EDuration="5.419038663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:05.415558625 +0000 UTC m=+2271.907375170" watchObservedRunningTime="2026-10-30 21:15:05.419038663 +0000 UTC m=+2271.910855200"
Oct 30 21:15:06.212027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.211995    4810 scope.go:122] "RemoveContainer" containerID="fa2c13c4ea2cc7185d6531036857e9424f9e60b0d7f4b04ccd7b3ff831b12d28"
Oct 30 21:15:06.212477 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.212407    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n_odh-ai-gateway-infra(a0e47096-ee02-476d-ab4c-d7d0846a976a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a"
Oct 30 21:15:06.214065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.214037    4810 generic.go:356] "Generic (PLEG): container finished" podID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerID="a1119e0608305dab35170f691deaef9a6daa59e114cbf1f6f9cc01b35098f798" exitCode=60
Oct 30 21:15:06.214221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.214135    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" event={"ID":"f91a2f39-f9cc-4bd9-8794-98d575ca8481","Type":"ContainerDied","Data":"a1119e0608305dab35170f691deaef9a6daa59e114cbf1f6f9cc01b35098f798"}
Oct 30 21:15:06.214221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.214201    4810 scope.go:122] "RemoveContainer" containerID="5e24f80f25dd5df8e4283546607ed9eb106853dc5ab928fdd875fffb267c7716"
Oct 30 21:15:06.214473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.214453    4810 scope.go:122] "RemoveContainer" containerID="a1119e0608305dab35170f691deaef9a6daa59e114cbf1f6f9cc01b35098f798"
Oct 30 21:15:06.214837 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.214811    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q_odh-ai-gateway-infra(f91a2f39-f9cc-4bd9-8794-98d575ca8481)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481"
Oct 30 21:15:06.217244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.217225    4810 scope.go:122] "RemoveContainer" containerID="6d1446f2726bc242398648a9e6c53e4a56a0f3ce0774c816da79bf6945e80546"
Oct 30 21:15:06.217557 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.217536    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg_odh-ai-gateway-infra(69815735-5039-4348-ae5d-68881b8555bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg" podUID="69815735-5039-4348-ae5d-68881b8555bd"
Oct 30 21:15:06.219233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.219180    4810 generic.go:356] "Generic (PLEG): container finished" podID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerID="bef57a34aefd550a45ea1912771ce800ebdb71a0b4b7ab5d16f2b6fe1b5ab7d3" exitCode=60
Oct 30 21:15:06.219345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.219265    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" event={"ID":"676f8fee-509a-4b6f-8fe0-ceaaae8f709a","Type":"ContainerDied","Data":"bef57a34aefd550a45ea1912771ce800ebdb71a0b4b7ab5d16f2b6fe1b5ab7d3"}
Oct 30 21:15:06.219473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.219457    4810 scope.go:122] "RemoveContainer" containerID="bef57a34aefd550a45ea1912771ce800ebdb71a0b4b7ab5d16f2b6fe1b5ab7d3"
Oct 30 21:15:06.219785 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.219728    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt_odh-ai-gateway-infra(676f8fee-509a-4b6f-8fe0-ceaaae8f709a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a"
Oct 30 21:15:06.221435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.221413    4810 scope.go:122] "RemoveContainer" containerID="ca89de7a4227267e5347099eaa0ba9ece425f6be15a762595d8d51a02d8e2e3d"
Oct 30 21:15:06.221710 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.221687    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g_odh-ai-gateway-infra(2d1184f0-d306-4de0-a4f3-016c48e5f897)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897"
Oct 30 21:15:06.223033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.223013    4810 scope.go:122] "RemoveContainer" containerID="72fe7ba4d4a56a843fffa0e72bc0cd329c5d21d25237f7c8de618382b8efc893"
Oct 30 21:15:06.223305 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.223284    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts_odh-ai-gateway-infra(def80bf3-e036-460e-9543-114a136c81fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts" podUID="def80bf3-e036-460e-9543-114a136c81fe"
Oct 30 21:15:06.224776 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.224758    4810 generic.go:356] "Generic (PLEG): container finished" podID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerID="d833d7f74526bdb84d58d45063bab735a8fc861c7469b7cd29555de038ee88b8" exitCode=60
Oct 30 21:15:06.224879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.224836    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" event={"ID":"c29ca7de-8f46-4550-9e3b-58e215775d2d","Type":"ContainerDied","Data":"d833d7f74526bdb84d58d45063bab735a8fc861c7469b7cd29555de038ee88b8"}
Oct 30 21:15:06.225139 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.225121    4810 scope.go:122] "RemoveContainer" containerID="d833d7f74526bdb84d58d45063bab735a8fc861c7469b7cd29555de038ee88b8"
Oct 30 21:15:06.225426 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.225404    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24_odh-ai-gateway-infra(c29ca7de-8f46-4550-9e3b-58e215775d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d"
Oct 30 21:15:06.226717 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.226694    4810 generic.go:356] "Generic (PLEG): container finished" podID="84174e23-a505-4963-a57e-41762b1faec7" containerID="75b9b8c75f8bef25a355d5b56c3762626dc058f19d6c55aa58f137c7c9541de4" exitCode=60
Oct 30 21:15:06.226803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.226724    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" event={"ID":"84174e23-a505-4963-a57e-41762b1faec7","Type":"ContainerDied","Data":"75b9b8c75f8bef25a355d5b56c3762626dc058f19d6c55aa58f137c7c9541de4"}
Oct 30 21:15:06.226974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.226960    4810 scope.go:122] "RemoveContainer" containerID="75b9b8c75f8bef25a355d5b56c3762626dc058f19d6c55aa58f137c7c9541de4"
Oct 30 21:15:06.227185 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.227168    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr_odh-ai-gateway-infra(84174e23-a505-4963-a57e-41762b1faec7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" podUID="84174e23-a505-4963-a57e-41762b1faec7"
Oct 30 21:15:06.228611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.228594    4810 scope.go:122] "RemoveContainer" containerID="5414ac07170ad8ea24410be54c8639512330b104d5b07331331ddc63a76dba02"
Oct 30 21:15:06.228828 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.228806    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm_odh-ai-gateway-infra(fb4d6041-3a3c-4980-9f2f-849b4bb7d457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457"
Oct 30 21:15:06.230148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.230121    4810 generic.go:356] "Generic (PLEG): container finished" podID="774fc560-ba2d-4285-8142-be854b22ffc6" containerID="1bfb8df3a5bde0241ae1b869be225a5a39360ca0cc30dc9b44c7688384bbc26a" exitCode=60
Oct 30 21:15:06.230248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.230141    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" event={"ID":"774fc560-ba2d-4285-8142-be854b22ffc6","Type":"ContainerDied","Data":"1bfb8df3a5bde0241ae1b869be225a5a39360ca0cc30dc9b44c7688384bbc26a"}
Oct 30 21:15:06.231358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.230913    4810 scope.go:122] "RemoveContainer" containerID="c0645065290f70ccfe605f8816552eb6de551316e1bb9b927e291523b16d9ef1"
Oct 30 21:15:06.231358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.231134    4810 scope.go:122] "RemoveContainer" containerID="1bfb8df3a5bde0241ae1b869be225a5a39360ca0cc30dc9b44c7688384bbc26a"
Oct 30 21:15:06.231493 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.231388    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9_odh-ai-gateway-infra(774fc560-ba2d-4285-8142-be854b22ffc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" podUID="774fc560-ba2d-4285-8142-be854b22ffc6"
Oct 30 21:15:06.232686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.232667    4810 generic.go:356] "Generic (PLEG): container finished" podID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerID="4fbd63963785dacfc8cc987281c9809eacf09a631e7f8e5bc00ade17824062ed" exitCode=60
Oct 30 21:15:06.232764 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.232733    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" event={"ID":"80ae6ddf-42ce-46fb-bae7-9facca564fbf","Type":"ContainerDied","Data":"4fbd63963785dacfc8cc987281c9809eacf09a631e7f8e5bc00ade17824062ed"}
Oct 30 21:15:06.232995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.232978    4810 scope.go:122] "RemoveContainer" containerID="4fbd63963785dacfc8cc987281c9809eacf09a631e7f8e5bc00ade17824062ed"
Oct 30 21:15:06.233222 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.233201    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r_odh-ai-gateway-infra(80ae6ddf-42ce-46fb-bae7-9facca564fbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf"
Oct 30 21:15:06.234730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.234708    4810 generic.go:356] "Generic (PLEG): container finished" podID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerID="d921e532e3d34872443ff649a36b8960c169963a5516d315bbc12e05bc94a287" exitCode=60
Oct 30 21:15:06.234813 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.234784    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" event={"ID":"e9df251a-ccad-45c9-a477-2fa0e8f67060","Type":"ContainerDied","Data":"d921e532e3d34872443ff649a36b8960c169963a5516d315bbc12e05bc94a287"}
Oct 30 21:15:06.234980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.234963    4810 scope.go:122] "RemoveContainer" containerID="d921e532e3d34872443ff649a36b8960c169963a5516d315bbc12e05bc94a287"
Oct 30 21:15:06.235162 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.235145    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj_odh-ai-gateway-infra(e9df251a-ccad-45c9-a477-2fa0e8f67060)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060"
Oct 30 21:15:06.236913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.236892    4810 generic.go:356] "Generic (PLEG): container finished" podID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerID="2cc43f432f6bc6247d2cd904e296b5549f7f733b240e75bf358236f37e7a8c10" exitCode=60
Oct 30 21:15:06.237002 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.236987    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" event={"ID":"3328918f-c5a6-4562-acbe-7d0f10ec1329","Type":"ContainerDied","Data":"2cc43f432f6bc6247d2cd904e296b5549f7f733b240e75bf358236f37e7a8c10"}
Oct 30 21:15:06.237186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.237173    4810 scope.go:122] "RemoveContainer" containerID="2cc43f432f6bc6247d2cd904e296b5549f7f733b240e75bf358236f37e7a8c10"
Oct 30 21:15:06.237383 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.237365    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-6wv5s_odh-ai-gateway-infra(3328918f-c5a6-4562-acbe-7d0f10ec1329)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329"
Oct 30 21:15:06.238851 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.238828    4810 generic.go:356] "Generic (PLEG): container finished" podID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerID="e9c7b2bbde0b5738a7198c0efe5df2d5fa20a1e2138274bec869420f613d370d" exitCode=60
Oct 30 21:15:06.238930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.238884    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" event={"ID":"fe530dfb-4534-49d6-91d3-28f198c77d88","Type":"ContainerDied","Data":"e9c7b2bbde0b5738a7198c0efe5df2d5fa20a1e2138274bec869420f613d370d"}
Oct 30 21:15:06.239090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.239076    4810 scope.go:122] "RemoveContainer" containerID="e9c7b2bbde0b5738a7198c0efe5df2d5fa20a1e2138274bec869420f613d370d"
Oct 30 21:15:06.239281 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.239265    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66_odh-ai-gateway-infra(fe530dfb-4534-49d6-91d3-28f198c77d88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88"
Oct 30 21:15:06.240825 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.240793    4810 scope.go:122] "RemoveContainer" containerID="6111b830195345e2370d9a1712847a4f30e59a264571b88643cd2125c888c300"
Oct 30 21:15:06.241070 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:06.241049    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629_odh-ai-gateway-infra(18183221-a049-4171-add0-1810e4939630)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629" podUID="18183221-a049-4171-add0-1810e4939630"
Oct 30 21:15:06.246754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.246734    4810 scope.go:122] "RemoveContainer" containerID="5dfa9fc7b1eaa75255b46fc44e60d40a8d296fd7cfd7c785c45b031477909bd7"
Oct 30 21:15:06.261327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.261311    4810 scope.go:122] "RemoveContainer" containerID="d94944382112dca94bd1a6e147bf8721d178b4fa7553f23d82ce0d538d5f20cf"
Oct 30 21:15:06.282041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.281382    4810 scope.go:122] "RemoveContainer" containerID="3f55ccc1ea3d658004075b0490aba41ad2d8c2662ce5c00348afe802d2b91cc3"
Oct 30 21:15:06.298673 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.298652    4810 scope.go:122] "RemoveContainer" containerID="9d11d5a27f5b9d6f166cd1eac5a19b2dc953bb53c715b1c4fccad25af4807c3d"
Oct 30 21:15:06.315350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.315328    4810 scope.go:122] "RemoveContainer" containerID="2cff708ec19e14a3bd10f4d36b5091711f0b15a817ee43a7bd4dbcfaf368db73"
Oct 30 21:15:06.345935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.337405    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"]
Oct 30 21:15:06.346498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.346474    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.356900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.353872    4810 scope.go:122] "RemoveContainer" containerID="15314da0385532fe3aaae419491e7d56470b842089842f1a7fd546996c9d659f"
Oct 30 21:15:06.357666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.357643    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"]
Oct 30 21:15:06.383247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.382635    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.383247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.382707    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.383247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.382729    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fpsp\" (UniqueName: \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-kube-api-access-8fpsp\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.383247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.382757    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.383247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.382829    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.383247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.382876    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.383247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.382904    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.383247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.382955    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.383247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.382986    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.400253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.400209    4810 scope.go:122] "RemoveContainer" containerID="23d3daff315934babaeb8e241ccb4daca8a02fb009d5f49bcbbffa6415cc9f1a"
Oct 30 21:15:06.426248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.426192    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-345bc4-openshift-default-dockercfg-fck6v\""
Oct 30 21:15:06.484176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484084    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484140    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484176    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484447 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484244    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484447 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484291    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484451    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484538    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484564    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fpsp\" (UniqueName: \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-kube-api-access-8fpsp\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484609    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484698    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484783    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.484953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484933    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.485041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.484968    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.485225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.485203    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.487077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.487044    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.487192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.487149    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.492933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.492911    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.495212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.495179    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fpsp\" (UniqueName: \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-kube-api-access-8fpsp\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.676173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.676141    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:06.986892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.981063    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"]
Oct 30 21:15:06.986892 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:06.982678    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1c5bd7_0550_4b20_9fa6_e5275b88253c.slice/crio-2264eb2bb51f7ad9e2edf0f409c54a6a2e835923b2ecb771c12d53a0e294c359 WatchSource:0}: Error finding container 2264eb2bb51f7ad9e2edf0f409c54a6a2e835923b2ecb771c12d53a0e294c359: Status 404 returned error can't find the container with id 2264eb2bb51f7ad9e2edf0f409c54a6a2e835923b2ecb771c12d53a0e294c359
Oct 30 21:15:06.986892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.986393    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:06.986892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.986473    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:06.986892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:06.986514    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:07.252751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.252722    4810 scope.go:122] "RemoveContainer" containerID="1bfb8df3a5bde0241ae1b869be225a5a39360ca0cc30dc9b44c7688384bbc26a"
Oct 30 21:15:07.253230 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:07.253115    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9_odh-ai-gateway-infra(774fc560-ba2d-4285-8142-be854b22ffc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" podUID="774fc560-ba2d-4285-8142-be854b22ffc6"
Oct 30 21:15:07.254524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.254503    4810 scope.go:122] "RemoveContainer" containerID="4fbd63963785dacfc8cc987281c9809eacf09a631e7f8e5bc00ade17824062ed"
Oct 30 21:15:07.254915 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:07.254888    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r_odh-ai-gateway-infra(80ae6ddf-42ce-46fb-bae7-9facca564fbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf"
Oct 30 21:15:07.256419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.256400    4810 scope.go:122] "RemoveContainer" containerID="d921e532e3d34872443ff649a36b8960c169963a5516d315bbc12e05bc94a287"
Oct 30 21:15:07.256700 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:07.256667    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj_odh-ai-gateway-infra(e9df251a-ccad-45c9-a477-2fa0e8f67060)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060"
Oct 30 21:15:07.258276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.258254    4810 scope.go:122] "RemoveContainer" containerID="2cc43f432f6bc6247d2cd904e296b5549f7f733b240e75bf358236f37e7a8c10"
Oct 30 21:15:07.258570 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:07.258545    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-6wv5s_odh-ai-gateway-infra(3328918f-c5a6-4562-acbe-7d0f10ec1329)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329"
Oct 30 21:15:07.259731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.259699    4810 scope.go:122] "RemoveContainer" containerID="e9c7b2bbde0b5738a7198c0efe5df2d5fa20a1e2138274bec869420f613d370d"
Oct 30 21:15:07.260009 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:07.259989    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66_odh-ai-gateway-infra(fe530dfb-4534-49d6-91d3-28f198c77d88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88"
Oct 30 21:15:07.261261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.261238    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" event={"ID":"ae1c5bd7-0550-4b20-9fa6-e5275b88253c","Type":"ContainerStarted","Data":"3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e"}
Oct 30 21:15:07.261346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.261289    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" event={"ID":"ae1c5bd7-0550-4b20-9fa6-e5275b88253c","Type":"ContainerStarted","Data":"2264eb2bb51f7ad9e2edf0f409c54a6a2e835923b2ecb771c12d53a0e294c359"}
Oct 30 21:15:07.263077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.263060    4810 scope.go:122] "RemoveContainer" containerID="a1119e0608305dab35170f691deaef9a6daa59e114cbf1f6f9cc01b35098f798"
Oct 30 21:15:07.263278 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:07.263258    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q_odh-ai-gateway-infra(f91a2f39-f9cc-4bd9-8794-98d575ca8481)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481"
Oct 30 21:15:07.264637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.264620    4810 scope.go:122] "RemoveContainer" containerID="bef57a34aefd550a45ea1912771ce800ebdb71a0b4b7ab5d16f2b6fe1b5ab7d3"
Oct 30 21:15:07.264816 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:07.264797    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt_odh-ai-gateway-infra(676f8fee-509a-4b6f-8fe0-ceaaae8f709a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a"
Oct 30 21:15:07.266354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.266307    4810 scope.go:122] "RemoveContainer" containerID="d833d7f74526bdb84d58d45063bab735a8fc861c7469b7cd29555de038ee88b8"
Oct 30 21:15:07.266523 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:07.266505    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24_odh-ai-gateway-infra(c29ca7de-8f46-4550-9e3b-58e215775d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d"
Oct 30 21:15:07.267983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.267966    4810 scope.go:122] "RemoveContainer" containerID="75b9b8c75f8bef25a355d5b56c3762626dc058f19d6c55aa58f137c7c9541de4"
Oct 30 21:15:07.268168 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:07.268151    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr_odh-ai-gateway-infra(84174e23-a505-4963-a57e-41762b1faec7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" podUID="84174e23-a505-4963-a57e-41762b1faec7"
Oct 30 21:15:07.348509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.348459    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" podStartSLOduration=1.348446724 podStartE2EDuration="1.348446724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:07.347744525 +0000 UTC m=+2273.839561071" watchObservedRunningTime="2026-10-30 21:15:07.348446724 +0000 UTC m=+2273.840263263"
Oct 30 21:15:07.678887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:07.676944    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:08.679887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:08.678618    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 21:15:08.679887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:08.678681    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" podUID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 21:15:09.686880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:09.684255    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 21:15:09.686880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:09.684332    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" podUID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 21:15:10.682487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:10.682449    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 21:15:10.682781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:10.682753    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" podUID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 21:15:11.684885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:11.680335    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 21:15:11.684885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:11.680410    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" podUID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 21:15:12.681881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.676548    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused" start-of-body=
Oct 30 21:15:12.681881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.676611    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" podUID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.39:15021/healthz/ready\": dial tcp 10.132.1.39:15021: connect: connection refused"
Oct 30 21:15:12.747467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.747428    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"]
Oct 30 21:15:12.752749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.752725    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.769078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.769045    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"]
Oct 30 21:15:12.846977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.846938    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.847344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.847317    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bmb\" (UniqueName: \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-kube-api-access-g9bmb\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.847540 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.847521    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.847775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.847737    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.949140 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.949057    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.949296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.949175    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bmb\" (UniqueName: \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-kube-api-access-g9bmb\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.949296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.949226    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.949382 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.949334    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.949920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.949891    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.950038 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.949936    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.951603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.951585    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:12.957827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:12.957808    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bmb\" (UniqueName: \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-kube-api-access-g9bmb\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:13.124765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:13.124727    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:13.416273 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:13.416237    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c8f50b_9264_4eed_8bce_d04d69583004.slice/crio-86e649b4d1811b53c59a85f7c2789a17b1f383b6cf489adb0d393015967ef23a WatchSource:0}: Error finding container 86e649b4d1811b53c59a85f7c2789a17b1f383b6cf489adb0d393015967ef23a: Status 404 returned error can't find the container with id 86e649b4d1811b53c59a85f7c2789a17b1f383b6cf489adb0d393015967ef23a
Oct 30 21:15:13.417511 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:13.417485    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"]
Oct 30 21:15:13.683543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:13.683506    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:13.684192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:13.684162    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:13.685070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:13.685054    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:15:14.397333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:14.397292    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8" event={"ID":"67c8f50b-9264-4eed-8bce-d04d69583004","Type":"ContainerStarted","Data":"abd76e44455d7633797b880f76987d3a5ed6e13657ac345bd5936b82eaaef410"}
Oct 30 21:15:14.397839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:14.397355    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8" event={"ID":"67c8f50b-9264-4eed-8bce-d04d69583004","Type":"ContainerStarted","Data":"86e649b4d1811b53c59a85f7c2789a17b1f383b6cf489adb0d393015967ef23a"}
Oct 30 21:15:14.397839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:14.397626    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:14.419522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:14.419469    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8" podStartSLOduration=2.419456411 podStartE2EDuration="2.419456411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:14.418836754 +0000 UTC m=+2280.910653311" watchObservedRunningTime="2026-10-30 21:15:14.419456411 +0000 UTC m=+2280.911272970"
Oct 30 21:15:15.667887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:15.657742    4810 scope.go:122] "RemoveContainer" containerID="5414ac07170ad8ea24410be54c8639512330b104d5b07331331ddc63a76dba02"
Oct 30 21:15:15.667887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:15.658784    4810 scope.go:122] "RemoveContainer" containerID="6111b830195345e2370d9a1712847a4f30e59a264571b88643cd2125c888c300"
Oct 30 21:15:15.667887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:15.659551    4810 scope.go:122] "RemoveContainer" containerID="fa2c13c4ea2cc7185d6531036857e9424f9e60b0d7f4b04ccd7b3ff831b12d28"
Oct 30 21:15:16.427286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:16.427249    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm" event={"ID":"fb4d6041-3a3c-4980-9f2f-849b4bb7d457","Type":"ContainerStarted","Data":"318681ace3c43eae15df1182d8dee01760fc190744d9cf112cf17caa847c2903"}
Oct 30 21:15:16.429365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:16.429340    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629" event={"ID":"18183221-a049-4171-add0-1810e4939630","Type":"ContainerStarted","Data":"ef80c1342cd47ce211fdecff81762da11e3710aa0f75e2f36e869e993da3dd8c"}
Oct 30 21:15:16.431264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:16.431244    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" event={"ID":"a0e47096-ee02-476d-ab4c-d7d0846a976a","Type":"ContainerStarted","Data":"ee205b8f18e702c645d14b0b61931e577abecda61f893f692b67e86000f67712"}
Oct 30 21:15:16.453670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:16.453628    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm" podStartSLOduration=16.453616667 podStartE2EDuration="16.453616667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:16.449561595 +0000 UTC m=+2282.941378140" watchObservedRunningTime="2026-10-30 21:15:16.453616667 +0000 UTC m=+2282.945433211"
Oct 30 21:15:16.654470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:16.654430    4810 scope.go:122] "RemoveContainer" containerID="d921e532e3d34872443ff649a36b8960c169963a5516d315bbc12e05bc94a287"
Oct 30 21:15:16.654889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:16.654831    4810 scope.go:122] "RemoveContainer" containerID="6d1446f2726bc242398648a9e6c53e4a56a0f3ce0774c816da79bf6945e80546"
Oct 30 21:15:16.655009 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:16.654936    4810 scope.go:122] "RemoveContainer" containerID="2cc43f432f6bc6247d2cd904e296b5549f7f733b240e75bf358236f37e7a8c10"
Oct 30 21:15:17.018396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.018364    4810 scope.go:122] "RemoveContainer" containerID="56e58a906c821c0c62c435f37675eab6a654d5aa92c2d5891f3e90776fb194df"
Oct 30 21:15:17.110042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.110009    4810 scope.go:122] "RemoveContainer" containerID="2cc3646a85962af5aafd9601d1c8ca10cff1b0d9e155bee4aed6c5a36f6d7fce"
Oct 30 21:15:17.441542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.441461    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" event={"ID":"e9df251a-ccad-45c9-a477-2fa0e8f67060","Type":"ContainerStarted","Data":"b12c617cccc9c7d6b0319252f7b2d65f3b9b9a0b8b595aa2d73a46e8be67f62c"}
Oct 30 21:15:17.443489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.443458    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" event={"ID":"3328918f-c5a6-4562-acbe-7d0f10ec1329","Type":"ContainerStarted","Data":"b4caf60e928f4f90641ef44435a417746d9437acad76214800e8cb5c5aed6ff2"}
Oct 30 21:15:17.445262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.445239    4810 generic.go:356] "Generic (PLEG): container finished" podID="18183221-a049-4171-add0-1810e4939630" containerID="ef80c1342cd47ce211fdecff81762da11e3710aa0f75e2f36e869e993da3dd8c" exitCode=60
Oct 30 21:15:17.445335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.445305    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629" event={"ID":"18183221-a049-4171-add0-1810e4939630","Type":"ContainerDied","Data":"ef80c1342cd47ce211fdecff81762da11e3710aa0f75e2f36e869e993da3dd8c"}
Oct 30 21:15:17.445395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.445364    4810 scope.go:122] "RemoveContainer" containerID="6111b830195345e2370d9a1712847a4f30e59a264571b88643cd2125c888c300"
Oct 30 21:15:17.445700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.445683    4810 scope.go:122] "RemoveContainer" containerID="ef80c1342cd47ce211fdecff81762da11e3710aa0f75e2f36e869e993da3dd8c"
Oct 30 21:15:17.446028 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:17.446005    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629_odh-ai-gateway-infra(18183221-a049-4171-add0-1810e4939630)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629" podUID="18183221-a049-4171-add0-1810e4939630"
Oct 30 21:15:17.447423 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.447401    4810 generic.go:356] "Generic (PLEG): container finished" podID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerID="ee205b8f18e702c645d14b0b61931e577abecda61f893f692b67e86000f67712" exitCode=60
Oct 30 21:15:17.447524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.447468    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" event={"ID":"a0e47096-ee02-476d-ab4c-d7d0846a976a","Type":"ContainerDied","Data":"ee205b8f18e702c645d14b0b61931e577abecda61f893f692b67e86000f67712"}
Oct 30 21:15:17.447834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.447814    4810 scope.go:122] "RemoveContainer" containerID="ee205b8f18e702c645d14b0b61931e577abecda61f893f692b67e86000f67712"
Oct 30 21:15:17.448154 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:17.448132    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n_odh-ai-gateway-infra(a0e47096-ee02-476d-ab4c-d7d0846a976a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a"
Oct 30 21:15:17.449440 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.449421    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg" event={"ID":"69815735-5039-4348-ae5d-68881b8555bd","Type":"ContainerStarted","Data":"edc0c1a808dac6e4315516c7e6ecaa9bdf2f7ab25d81aa3b6915f6fc4a2ea5e3"}
Oct 30 21:15:17.451358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.451339    4810 generic.go:356] "Generic (PLEG): container finished" podID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerID="318681ace3c43eae15df1182d8dee01760fc190744d9cf112cf17caa847c2903" exitCode=60
Oct 30 21:15:17.451427 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.451378    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm" event={"ID":"fb4d6041-3a3c-4980-9f2f-849b4bb7d457","Type":"ContainerDied","Data":"318681ace3c43eae15df1182d8dee01760fc190744d9cf112cf17caa847c2903"}
Oct 30 21:15:17.451673 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.451660    4810 scope.go:122] "RemoveContainer" containerID="318681ace3c43eae15df1182d8dee01760fc190744d9cf112cf17caa847c2903"
Oct 30 21:15:17.451904 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:17.451887    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm_odh-ai-gateway-infra(fb4d6041-3a3c-4980-9f2f-849b4bb7d457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457"
Oct 30 21:15:17.476958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.472044    4810 scope.go:122] "RemoveContainer" containerID="fa2c13c4ea2cc7185d6531036857e9424f9e60b0d7f4b04ccd7b3ff831b12d28"
Oct 30 21:15:17.512877 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.510626    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629" podStartSLOduration=17.510610373 podStartE2EDuration="17.510610373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:16.513909499 +0000 UTC m=+2283.005726045" watchObservedRunningTime="2026-10-30 21:15:17.510610373 +0000 UTC m=+2284.002426918"
Oct 30 21:15:17.512877 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.512660    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"]
Oct 30 21:15:17.544683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.544649    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"]
Oct 30 21:15:17.546997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.546974    4810 scope.go:122] "RemoveContainer" containerID="5414ac07170ad8ea24410be54c8639512330b104d5b07331331ddc63a76dba02"
Oct 30 21:15:17.560670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.560639    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"]
Oct 30 21:15:17.580391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.580329    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg" podStartSLOduration=17.580311489 podStartE2EDuration="17.580311489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:17.569373367 +0000 UTC m=+2284.061189944" watchObservedRunningTime="2026-10-30 21:15:17.580311489 +0000 UTC m=+2284.072128038"
Oct 30 21:15:17.664898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.664872    4810 scope.go:122] "RemoveContainer" containerID="4fbd63963785dacfc8cc987281c9809eacf09a631e7f8e5bc00ade17824062ed"
Oct 30 21:15:17.665069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.664992    4810 scope.go:122] "RemoveContainer" containerID="ca89de7a4227267e5347099eaa0ba9ece425f6be15a762595d8d51a02d8e2e3d"
Oct 30 21:15:17.665131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:17.665116    4810 scope.go:122] "RemoveContainer" containerID="e9c7b2bbde0b5738a7198c0efe5df2d5fa20a1e2138274bec869420f613d370d"
Oct 30 21:15:18.466898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.466839    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" event={"ID":"80ae6ddf-42ce-46fb-bae7-9facca564fbf","Type":"ContainerStarted","Data":"9774b44a3068e87e471e34e0a7c23509d5b3856cec4dfad8daac6aba1429e5d2"}
Oct 30 21:15:18.469628 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.469601    4810 generic.go:356] "Generic (PLEG): container finished" podID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerID="b12c617cccc9c7d6b0319252f7b2d65f3b9b9a0b8b595aa2d73a46e8be67f62c" exitCode=60
Oct 30 21:15:18.469752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.469666    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" event={"ID":"e9df251a-ccad-45c9-a477-2fa0e8f67060","Type":"ContainerDied","Data":"b12c617cccc9c7d6b0319252f7b2d65f3b9b9a0b8b595aa2d73a46e8be67f62c"}
Oct 30 21:15:18.469752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.469717    4810 scope.go:122] "RemoveContainer" containerID="d921e532e3d34872443ff649a36b8960c169963a5516d315bbc12e05bc94a287"
Oct 30 21:15:18.470065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.470049    4810 scope.go:122] "RemoveContainer" containerID="b12c617cccc9c7d6b0319252f7b2d65f3b9b9a0b8b595aa2d73a46e8be67f62c"
Oct 30 21:15:18.470359 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:18.470341    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj_odh-ai-gateway-infra(e9df251a-ccad-45c9-a477-2fa0e8f67060)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060"
Oct 30 21:15:18.477919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.471846    4810 generic.go:356] "Generic (PLEG): container finished" podID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerID="b4caf60e928f4f90641ef44435a417746d9437acad76214800e8cb5c5aed6ff2" exitCode=60
Oct 30 21:15:18.477919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.471964    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" event={"ID":"3328918f-c5a6-4562-acbe-7d0f10ec1329","Type":"ContainerDied","Data":"b4caf60e928f4f90641ef44435a417746d9437acad76214800e8cb5c5aed6ff2"}
Oct 30 21:15:18.477919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.472289    4810 scope.go:122] "RemoveContainer" containerID="b4caf60e928f4f90641ef44435a417746d9437acad76214800e8cb5c5aed6ff2"
Oct 30 21:15:18.477919 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:18.472748    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-6wv5s_odh-ai-gateway-infra(3328918f-c5a6-4562-acbe-7d0f10ec1329)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329"
Oct 30 21:15:18.477919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.475279    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" event={"ID":"fe530dfb-4534-49d6-91d3-28f198c77d88","Type":"ContainerStarted","Data":"562560ddebe88225b444f459213c4ca737022e0cd328f3916266b134129524f3"}
Oct 30 21:15:18.480340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.480319    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"]
Oct 30 21:15:18.483043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.483019    4810 generic.go:356] "Generic (PLEG): container finished" podID="69815735-5039-4348-ae5d-68881b8555bd" containerID="edc0c1a808dac6e4315516c7e6ecaa9bdf2f7ab25d81aa3b6915f6fc4a2ea5e3" exitCode=60
Oct 30 21:15:18.483767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.483702    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg" event={"ID":"69815735-5039-4348-ae5d-68881b8555bd","Type":"ContainerDied","Data":"edc0c1a808dac6e4315516c7e6ecaa9bdf2f7ab25d81aa3b6915f6fc4a2ea5e3"}
Oct 30 21:15:18.484354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.484337    4810 scope.go:122] "RemoveContainer" containerID="edc0c1a808dac6e4315516c7e6ecaa9bdf2f7ab25d81aa3b6915f6fc4a2ea5e3"
Oct 30 21:15:18.484771 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:18.484749    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg_odh-ai-gateway-infra(69815735-5039-4348-ae5d-68881b8555bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg" podUID="69815735-5039-4348-ae5d-68881b8555bd"
Oct 30 21:15:18.491942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.487528    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g" event={"ID":"2d1184f0-d306-4de0-a4f3-016c48e5f897","Type":"ContainerStarted","Data":"3014753c5c49209932ecb12060745609c53411b8d2e7c5c7ab759cce527c36ea"}
Oct 30 21:15:18.494034 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.494015    4810 scope.go:122] "RemoveContainer" containerID="2cc43f432f6bc6247d2cd904e296b5549f7f733b240e75bf358236f37e7a8c10"
Oct 30 21:15:18.534259 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.534229    4810 scope.go:122] "RemoveContainer" containerID="6d1446f2726bc242398648a9e6c53e4a56a0f3ce0774c816da79bf6945e80546"
Oct 30 21:15:18.535069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.535037    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"]
Oct 30 21:15:18.609884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.609586    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"]
Oct 30 21:15:18.655192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.654470    4810 scope.go:122] "RemoveContainer" containerID="1bfb8df3a5bde0241ae1b869be225a5a39360ca0cc30dc9b44c7688384bbc26a"
Oct 30 21:15:18.655192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.654751    4810 scope.go:122] "RemoveContainer" containerID="bef57a34aefd550a45ea1912771ce800ebdb71a0b4b7ab5d16f2b6fe1b5ab7d3"
Oct 30 21:15:18.655394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.655286    4810 scope.go:122] "RemoveContainer" containerID="75b9b8c75f8bef25a355d5b56c3762626dc058f19d6c55aa58f137c7c9541de4"
Oct 30 21:15:18.692486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.692433    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g" podStartSLOduration=18.692417372 podStartE2EDuration="18.692417372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:18.689326577 +0000 UTC m=+2285.181143124" watchObservedRunningTime="2026-10-30 21:15:18.692417372 +0000 UTC m=+2285.184233919"
Oct 30 21:15:18.783147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.782870    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"
Oct 30 21:15:18.847380 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.847204    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"
Oct 30 21:15:18.852310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.852281    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"
Oct 30 21:15:18.957255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.957213    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18183221-a049-4171-add0-1810e4939630-kube-api-access-t7pxg\" (UniqueName: \"kubernetes.io/projected/18183221-a049-4171-add0-1810e4939630-kube-api-access-t7pxg\") pod \"18183221-a049-4171-add0-1810e4939630\" (UID: \"18183221-a049-4171-add0-1810e4939630\") "
Oct 30 21:15:18.957435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.957371    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-kube-api-access-hq879\" (UniqueName: \"kubernetes.io/projected/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-kube-api-access-hq879\") pod \"fb4d6041-3a3c-4980-9f2f-849b4bb7d457\" (UID: \"fb4d6041-3a3c-4980-9f2f-849b4bb7d457\") "
Oct 30 21:15:18.957541 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.957506    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-maas-api-service-ca\") pod \"fb4d6041-3a3c-4980-9f2f-849b4bb7d457\" (UID: \"fb4d6041-3a3c-4980-9f2f-849b4bb7d457\") "
Oct 30 21:15:18.957882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.957537    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18183221-a049-4171-add0-1810e4939630-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18183221-a049-4171-add0-1810e4939630-maas-api-service-ca\") pod \"18183221-a049-4171-add0-1810e4939630\" (UID: \"18183221-a049-4171-add0-1810e4939630\") "
Oct 30 21:15:18.957984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.957834    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-maas-api-service-ca" pod "fb4d6041-3a3c-4980-9f2f-849b4bb7d457" (UID: "fb4d6041-3a3c-4980-9f2f-849b4bb7d457"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:18.957984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.957934    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18183221-a049-4171-add0-1810e4939630-maas-api-service-ca" pod "18183221-a049-4171-add0-1810e4939630" (UID: "18183221-a049-4171-add0-1810e4939630"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:18.958112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.957983    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e47096-ee02-476d-ab4c-d7d0846a976a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e47096-ee02-476d-ab4c-d7d0846a976a-maas-api-service-ca\") pod \"a0e47096-ee02-476d-ab4c-d7d0846a976a\" (UID: \"a0e47096-ee02-476d-ab4c-d7d0846a976a\") "
Oct 30 21:15:18.958112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.958008    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e47096-ee02-476d-ab4c-d7d0846a976a-kube-api-access-6bxgq\" (UniqueName: \"kubernetes.io/projected/a0e47096-ee02-476d-ab4c-d7d0846a976a-kube-api-access-6bxgq\") pod \"a0e47096-ee02-476d-ab4c-d7d0846a976a\" (UID: \"a0e47096-ee02-476d-ab4c-d7d0846a976a\") "
Oct 30 21:15:18.959097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.958353    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e47096-ee02-476d-ab4c-d7d0846a976a-maas-api-service-ca" pod "a0e47096-ee02-476d-ab4c-d7d0846a976a" (UID: "a0e47096-ee02-476d-ab4c-d7d0846a976a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:18.959097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.958795    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:18.959097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.958818    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18183221-a049-4171-add0-1810e4939630-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:18.959097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.958836    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e47096-ee02-476d-ab4c-d7d0846a976a-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:18.978039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.977694    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-kube-api-access-hq879" pod "fb4d6041-3a3c-4980-9f2f-849b4bb7d457" (UID: "fb4d6041-3a3c-4980-9f2f-849b4bb7d457"). InnerVolumeSpecName "kube-api-access-hq879". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:18.978260 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.978232    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e47096-ee02-476d-ab4c-d7d0846a976a-kube-api-access-6bxgq" pod "a0e47096-ee02-476d-ab4c-d7d0846a976a" (UID: "a0e47096-ee02-476d-ab4c-d7d0846a976a"). InnerVolumeSpecName "kube-api-access-6bxgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:18.981122 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:18.981086    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18183221-a049-4171-add0-1810e4939630-kube-api-access-t7pxg" pod "18183221-a049-4171-add0-1810e4939630" (UID: "18183221-a049-4171-add0-1810e4939630"). InnerVolumeSpecName "kube-api-access-t7pxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:19.060419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.060383    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxgq\" (UniqueName: \"kubernetes.io/projected/a0e47096-ee02-476d-ab4c-d7d0846a976a-kube-api-access-6bxgq\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.060419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.060413    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7pxg\" (UniqueName: \"kubernetes.io/projected/18183221-a049-4171-add0-1810e4939630-kube-api-access-t7pxg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.060419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.060423    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq879\" (UniqueName: \"kubernetes.io/projected/fb4d6041-3a3c-4980-9f2f-849b4bb7d457-kube-api-access-hq879\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.509904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.508727    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"
Oct 30 21:15:19.509904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.508759    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n" event={"ID":"a0e47096-ee02-476d-ab4c-d7d0846a976a","Type":"ContainerDied","Data":"1e4e338dcb970426a091797b585c46a080d9dbf0849b593fbc0d4bb624391757"}
Oct 30 21:15:19.509904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.508814    4810 scope.go:122] "RemoveContainer" containerID="ee205b8f18e702c645d14b0b61931e577abecda61f893f692b67e86000f67712"
Oct 30 21:15:19.518036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.517983    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"]
Oct 30 21:15:19.523605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.523575    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" event={"ID":"676f8fee-509a-4b6f-8fe0-ceaaae8f709a","Type":"ContainerStarted","Data":"46d2e17fac0f8aed4f5ce39de4a179499a1b7e7ee7fb51d907a2bfda78628e76"}
Oct 30 21:15:19.533126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.533102    4810 generic.go:356] "Generic (PLEG): container finished" podID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerID="3014753c5c49209932ecb12060745609c53411b8d2e7c5c7ab759cce527c36ea" exitCode=60
Oct 30 21:15:19.533236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.533218    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g" event={"ID":"2d1184f0-d306-4de0-a4f3-016c48e5f897","Type":"ContainerDied","Data":"3014753c5c49209932ecb12060745609c53411b8d2e7c5c7ab759cce527c36ea"}
Oct 30 21:15:19.533285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.533273    4810 scope.go:122] "RemoveContainer" containerID="ca89de7a4227267e5347099eaa0ba9ece425f6be15a762595d8d51a02d8e2e3d"
Oct 30 21:15:19.533682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.533658    4810 scope.go:122] "RemoveContainer" containerID="3014753c5c49209932ecb12060745609c53411b8d2e7c5c7ab759cce527c36ea"
Oct 30 21:15:19.534090 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:19.534067    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g_odh-ai-gateway-infra(2d1184f0-d306-4de0-a4f3-016c48e5f897)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897"
Oct 30 21:15:19.535667 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.535646    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" event={"ID":"84174e23-a505-4963-a57e-41762b1faec7","Type":"ContainerStarted","Data":"aab06f268172fe1cca243106f99a989f9f7075ecb1e0dd58b4552cb4efe4904d"}
Oct 30 21:15:19.537129 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.537110    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"
Oct 30 21:15:19.537229 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.537133    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm" event={"ID":"fb4d6041-3a3c-4980-9f2f-849b4bb7d457","Type":"ContainerDied","Data":"983e2dda42d2ae6f8d1595ec12ccd29e992fcf25008ab219068bc2ad437af51f"}
Oct 30 21:15:19.540472 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.540449    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" event={"ID":"774fc560-ba2d-4285-8142-be854b22ffc6","Type":"ContainerStarted","Data":"7f822171b73ffe509ed2ba93f814e8570e34d2c71a92f30ffb625f497378acf3"}
Oct 30 21:15:19.546057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.546025    4810 generic.go:356] "Generic (PLEG): container finished" podID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerID="9774b44a3068e87e471e34e0a7c23509d5b3856cec4dfad8daac6aba1429e5d2" exitCode=60
Oct 30 21:15:19.546204 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.546181    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" event={"ID":"80ae6ddf-42ce-46fb-bae7-9facca564fbf","Type":"ContainerDied","Data":"9774b44a3068e87e471e34e0a7c23509d5b3856cec4dfad8daac6aba1429e5d2"}
Oct 30 21:15:19.551609 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.551596    4810 scope.go:122] "RemoveContainer" containerID="318681ace3c43eae15df1182d8dee01760fc190744d9cf112cf17caa847c2903"
Oct 30 21:15:19.556915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.556205    4810 generic.go:356] "Generic (PLEG): container finished" podID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerID="562560ddebe88225b444f459213c4ca737022e0cd328f3916266b134129524f3" exitCode=60
Oct 30 21:15:19.556915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.556285    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" event={"ID":"fe530dfb-4534-49d6-91d3-28f198c77d88","Type":"ContainerDied","Data":"562560ddebe88225b444f459213c4ca737022e0cd328f3916266b134129524f3"}
Oct 30 21:15:19.556915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.556740    4810 scope.go:122] "RemoveContainer" containerID="562560ddebe88225b444f459213c4ca737022e0cd328f3916266b134129524f3"
Oct 30 21:15:19.563558 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:19.561423    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66_odh-ai-gateway-infra(fe530dfb-4534-49d6-91d3-28f198c77d88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88"
Oct 30 21:15:19.563558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.562630    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629" event={"ID":"18183221-a049-4171-add0-1810e4939630","Type":"ContainerDied","Data":"16dc02e520257c65fdcf76c67276e15c413ac3e53676c4a7682336962132f76c"}
Oct 30 21:15:19.563558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.562715    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"
Oct 30 21:15:19.595528 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.595494    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"]
Oct 30 21:15:19.600904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.600835    4810 scope.go:122] "RemoveContainer" containerID="4fbd63963785dacfc8cc987281c9809eacf09a631e7f8e5bc00ade17824062ed"
Oct 30 21:15:19.634463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.632614    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"]
Oct 30 21:15:19.640754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.640725    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a574c2-29889915-dkd6n"]
Oct 30 21:15:19.655994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.655816    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"]
Oct 30 21:15:19.658445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.658422    4810 scope.go:122] "RemoveContainer" containerID="a1119e0608305dab35170f691deaef9a6daa59e114cbf1f6f9cc01b35098f798"
Oct 30 21:15:19.660761 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.660740    4810 scope.go:122] "RemoveContainer" containerID="72fe7ba4d4a56a843fffa0e72bc0cd329c5d21d25237f7c8de618382b8efc893"
Oct 30 21:15:19.664643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.664622    4810 scope.go:122] "RemoveContainer" containerID="d833d7f74526bdb84d58d45063bab735a8fc861c7469b7cd29555de038ee88b8"
Oct 30 21:15:19.701561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.701418    4810 scope.go:122] "RemoveContainer" containerID="e9c7b2bbde0b5738a7198c0efe5df2d5fa20a1e2138274bec869420f613d370d"
Oct 30 21:15:19.780381 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.780320    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a" path="/var/lib/kubelet/pods/a0e47096-ee02-476d-ab4c-d7d0846a976a/volumes"
Oct 30 21:15:19.781200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.781163    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889915-c26vm"]
Oct 30 21:15:19.783151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.783133    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"]
Oct 30 21:15:19.903631 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.903574    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"]
Oct 30 21:15:19.916323 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.916123    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889915-8m629"]
Oct 30 21:15:19.922217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.922186    4810 scope.go:122] "RemoveContainer" containerID="ef80c1342cd47ce211fdecff81762da11e3710aa0f75e2f36e869e993da3dd8c"
Oct 30 21:15:19.973441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:19.973414    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"
Oct 30 21:15:20.113894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.110008    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69815735-5039-4348-ae5d-68881b8555bd-kube-api-access-95rdr\" (UniqueName: \"kubernetes.io/projected/69815735-5039-4348-ae5d-68881b8555bd-kube-api-access-95rdr\") pod \"69815735-5039-4348-ae5d-68881b8555bd\" (UID: \"69815735-5039-4348-ae5d-68881b8555bd\") "
Oct 30 21:15:20.113894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.110560    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69815735-5039-4348-ae5d-68881b8555bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69815735-5039-4348-ae5d-68881b8555bd-maas-api-service-ca\") pod \"69815735-5039-4348-ae5d-68881b8555bd\" (UID: \"69815735-5039-4348-ae5d-68881b8555bd\") "
Oct 30 21:15:20.113894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.111639    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69815735-5039-4348-ae5d-68881b8555bd-maas-api-service-ca" pod "69815735-5039-4348-ae5d-68881b8555bd" (UID: "69815735-5039-4348-ae5d-68881b8555bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.146469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.146439    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"
Oct 30 21:15:20.148634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.148610    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"
Oct 30 21:15:20.150841 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.150799    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"
Oct 30 21:15:20.212993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.212645    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69815735-5039-4348-ae5d-68881b8555bd-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.273348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.273308    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69815735-5039-4348-ae5d-68881b8555bd-kube-api-access-95rdr" pod "69815735-5039-4348-ae5d-68881b8555bd" (UID: "69815735-5039-4348-ae5d-68881b8555bd"). InnerVolumeSpecName "kube-api-access-95rdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.314017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.313986    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ae6ddf-42ce-46fb-bae7-9facca564fbf-kube-api-access-k8j2f\" (UniqueName: \"kubernetes.io/projected/80ae6ddf-42ce-46fb-bae7-9facca564fbf-kube-api-access-k8j2f\") pod \"80ae6ddf-42ce-46fb-bae7-9facca564fbf\" (UID: \"80ae6ddf-42ce-46fb-bae7-9facca564fbf\") "
Oct 30 21:15:20.314141 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.314058    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9df251a-ccad-45c9-a477-2fa0e8f67060-kube-api-access-jlqb2\" (UniqueName: \"kubernetes.io/projected/e9df251a-ccad-45c9-a477-2fa0e8f67060-kube-api-access-jlqb2\") pod \"e9df251a-ccad-45c9-a477-2fa0e8f67060\" (UID: \"e9df251a-ccad-45c9-a477-2fa0e8f67060\") "
Oct 30 21:15:20.314141 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.314093    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ae6ddf-42ce-46fb-bae7-9facca564fbf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ae6ddf-42ce-46fb-bae7-9facca564fbf-maas-api-service-ca\") pod \"80ae6ddf-42ce-46fb-bae7-9facca564fbf\" (UID: \"80ae6ddf-42ce-46fb-bae7-9facca564fbf\") "
Oct 30 21:15:20.314141 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.314134    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3328918f-c5a6-4562-acbe-7d0f10ec1329-kube-api-access-9fqnw\" (UniqueName: \"kubernetes.io/projected/3328918f-c5a6-4562-acbe-7d0f10ec1329-kube-api-access-9fqnw\") pod \"3328918f-c5a6-4562-acbe-7d0f10ec1329\" (UID: \"3328918f-c5a6-4562-acbe-7d0f10ec1329\") "
Oct 30 21:15:20.314329 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.314165    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9df251a-ccad-45c9-a477-2fa0e8f67060-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9df251a-ccad-45c9-a477-2fa0e8f67060-maas-api-service-ca\") pod \"e9df251a-ccad-45c9-a477-2fa0e8f67060\" (UID: \"e9df251a-ccad-45c9-a477-2fa0e8f67060\") "
Oct 30 21:15:20.314329 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.314212    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3328918f-c5a6-4562-acbe-7d0f10ec1329-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3328918f-c5a6-4562-acbe-7d0f10ec1329-maas-api-service-ca\") pod \"3328918f-c5a6-4562-acbe-7d0f10ec1329\" (UID: \"3328918f-c5a6-4562-acbe-7d0f10ec1329\") "
Oct 30 21:15:20.314760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.314732    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9df251a-ccad-45c9-a477-2fa0e8f67060-maas-api-service-ca" pod "e9df251a-ccad-45c9-a477-2fa0e8f67060" (UID: "e9df251a-ccad-45c9-a477-2fa0e8f67060"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.315041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.315018    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95rdr\" (UniqueName: \"kubernetes.io/projected/69815735-5039-4348-ae5d-68881b8555bd-kube-api-access-95rdr\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.315116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.315048    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9df251a-ccad-45c9-a477-2fa0e8f67060-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.315182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.315137    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3328918f-c5a6-4562-acbe-7d0f10ec1329-maas-api-service-ca" pod "3328918f-c5a6-4562-acbe-7d0f10ec1329" (UID: "3328918f-c5a6-4562-acbe-7d0f10ec1329"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.315267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.315239    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ae6ddf-42ce-46fb-bae7-9facca564fbf-maas-api-service-ca" pod "80ae6ddf-42ce-46fb-bae7-9facca564fbf" (UID: "80ae6ddf-42ce-46fb-bae7-9facca564fbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.316617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.316568    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3328918f-c5a6-4562-acbe-7d0f10ec1329-kube-api-access-9fqnw" pod "3328918f-c5a6-4562-acbe-7d0f10ec1329" (UID: "3328918f-c5a6-4562-acbe-7d0f10ec1329"). InnerVolumeSpecName "kube-api-access-9fqnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.317045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.317018    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9df251a-ccad-45c9-a477-2fa0e8f67060-kube-api-access-jlqb2" pod "e9df251a-ccad-45c9-a477-2fa0e8f67060" (UID: "e9df251a-ccad-45c9-a477-2fa0e8f67060"). InnerVolumeSpecName "kube-api-access-jlqb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.317175 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.317155    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ae6ddf-42ce-46fb-bae7-9facca564fbf-kube-api-access-k8j2f" pod "80ae6ddf-42ce-46fb-bae7-9facca564fbf" (UID: "80ae6ddf-42ce-46fb-bae7-9facca564fbf"). InnerVolumeSpecName "kube-api-access-k8j2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.416161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.416128    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8j2f\" (UniqueName: \"kubernetes.io/projected/80ae6ddf-42ce-46fb-bae7-9facca564fbf-kube-api-access-k8j2f\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.416161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.416158    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlqb2\" (UniqueName: \"kubernetes.io/projected/e9df251a-ccad-45c9-a477-2fa0e8f67060-kube-api-access-jlqb2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.416331 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.416167    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ae6ddf-42ce-46fb-bae7-9facca564fbf-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.416331 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.416176    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fqnw\" (UniqueName: \"kubernetes.io/projected/3328918f-c5a6-4562-acbe-7d0f10ec1329-kube-api-access-9fqnw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.416331 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.416186    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3328918f-c5a6-4562-acbe-7d0f10ec1329-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.573616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.573577    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"
Oct 30 21:15:20.574115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.573630    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg" event={"ID":"69815735-5039-4348-ae5d-68881b8555bd","Type":"ContainerDied","Data":"aa28115569b81e205fc9a8fd271f3fdd0b6582dc5c0d02aab8fa09608ed5c54c"}
Oct 30 21:15:20.574212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.574175    4810 scope.go:122] "RemoveContainer" containerID="edc0c1a808dac6e4315516c7e6ecaa9bdf2f7ab25d81aa3b6915f6fc4a2ea5e3"
Oct 30 21:15:20.578020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.577989    4810 generic.go:356] "Generic (PLEG): container finished" podID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerID="46d2e17fac0f8aed4f5ce39de4a179499a1b7e7ee7fb51d907a2bfda78628e76" exitCode=60
Oct 30 21:15:20.578230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.578063    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" event={"ID":"676f8fee-509a-4b6f-8fe0-ceaaae8f709a","Type":"ContainerDied","Data":"46d2e17fac0f8aed4f5ce39de4a179499a1b7e7ee7fb51d907a2bfda78628e76"}
Oct 30 21:15:20.578548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.578526    4810 scope.go:122] "RemoveContainer" containerID="46d2e17fac0f8aed4f5ce39de4a179499a1b7e7ee7fb51d907a2bfda78628e76"
Oct 30 21:15:20.578932 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:20.578907    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt_odh-ai-gateway-infra(676f8fee-509a-4b6f-8fe0-ceaaae8f709a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a"
Oct 30 21:15:20.589149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.587605    4810 generic.go:356] "Generic (PLEG): container finished" podID="84174e23-a505-4963-a57e-41762b1faec7" containerID="aab06f268172fe1cca243106f99a989f9f7075ecb1e0dd58b4552cb4efe4904d" exitCode=60
Oct 30 21:15:20.589149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.587729    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" event={"ID":"84174e23-a505-4963-a57e-41762b1faec7","Type":"ContainerDied","Data":"aab06f268172fe1cca243106f99a989f9f7075ecb1e0dd58b4552cb4efe4904d"}
Oct 30 21:15:20.589149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.588289    4810 scope.go:122] "RemoveContainer" containerID="aab06f268172fe1cca243106f99a989f9f7075ecb1e0dd58b4552cb4efe4904d"
Oct 30 21:15:20.589149 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:20.588660    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr_odh-ai-gateway-infra(84174e23-a505-4963-a57e-41762b1faec7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" podUID="84174e23-a505-4963-a57e-41762b1faec7"
Oct 30 21:15:20.592989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.592960    4810 generic.go:356] "Generic (PLEG): container finished" podID="774fc560-ba2d-4285-8142-be854b22ffc6" containerID="7f822171b73ffe509ed2ba93f814e8570e34d2c71a92f30ffb625f497378acf3" exitCode=60
Oct 30 21:15:20.593130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.593037    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" event={"ID":"774fc560-ba2d-4285-8142-be854b22ffc6","Type":"ContainerDied","Data":"7f822171b73ffe509ed2ba93f814e8570e34d2c71a92f30ffb625f497378acf3"}
Oct 30 21:15:20.593450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.593431    4810 scope.go:122] "RemoveContainer" containerID="7f822171b73ffe509ed2ba93f814e8570e34d2c71a92f30ffb625f497378acf3"
Oct 30 21:15:20.593799 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:20.593774    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9_odh-ai-gateway-infra(774fc560-ba2d-4285-8142-be854b22ffc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" podUID="774fc560-ba2d-4285-8142-be854b22ffc6"
Oct 30 21:15:20.594048 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.594028    4810 scope.go:122] "RemoveContainer" containerID="bef57a34aefd550a45ea1912771ce800ebdb71a0b4b7ab5d16f2b6fe1b5ab7d3"
Oct 30 21:15:20.596417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.596386    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r" event={"ID":"80ae6ddf-42ce-46fb-bae7-9facca564fbf","Type":"ContainerDied","Data":"16f505763732c629781cc949afd96fa85f3c9d4e85044c1a5916534ba97c9276"}
Oct 30 21:15:20.596513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.596455    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"
Oct 30 21:15:20.600273 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.600248    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"
Oct 30 21:15:20.600487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.600465    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj" event={"ID":"e9df251a-ccad-45c9-a477-2fa0e8f67060","Type":"ContainerDied","Data":"c1f9904c6eab3c3dfc500914f56bb59cd3fe4833c0c5bce99265159987ae8a44"}
Oct 30 21:15:20.610770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.610745    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"
Oct 30 21:15:20.610918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.610765    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s" event={"ID":"3328918f-c5a6-4562-acbe-7d0f10ec1329","Type":"ContainerDied","Data":"42ff05e06c3f501f747738a0343a82f27ba3c23cf6215d7a4cc9100894e15e35"}
Oct 30 21:15:20.642379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.642352    4810 scope.go:122] "RemoveContainer" containerID="75b9b8c75f8bef25a355d5b56c3762626dc058f19d6c55aa58f137c7c9541de4"
Oct 30 21:15:20.667644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.667620    4810 scope.go:122] "RemoveContainer" containerID="1bfb8df3a5bde0241ae1b869be225a5a39360ca0cc30dc9b44c7688384bbc26a"
Oct 30 21:15:20.692615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.692585    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"]
Oct 30 21:15:20.705709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.705678    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"]
Oct 30 21:15:20.716923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.714591    4810 scope.go:122] "RemoveContainer" containerID="9774b44a3068e87e471e34e0a7c23509d5b3856cec4dfad8daac6aba1429e5d2"
Oct 30 21:15:20.716923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.714791    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889915-jhmhg"]
Oct 30 21:15:20.738684 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.737914    4810 scope.go:122] "RemoveContainer" containerID="b12c617cccc9c7d6b0319252f7b2d65f3b9b9a0b8b595aa2d73a46e8be67f62c"
Oct 30 21:15:20.742689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.742001    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"]
Oct 30 21:15:20.748912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.748886    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-6wv5s"]
Oct 30 21:15:20.754469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.754417    4810 scope.go:122] "RemoveContainer" containerID="b4caf60e928f4f90641ef44435a417746d9437acad76214800e8cb5c5aed6ff2"
Oct 30 21:15:20.793527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.793490    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"]
Oct 30 21:15:20.807885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.805474    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"]
Oct 30 21:15:20.816316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.816287    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"
Oct 30 21:15:20.816937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.816885    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6741b1-29889915-pjq2r"]
Oct 30 21:15:20.825069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.824084    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1184f0-d306-4de0-a4f3-016c48e5f897-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1184f0-d306-4de0-a4f3-016c48e5f897-maas-api-service-ca\") pod \"2d1184f0-d306-4de0-a4f3-016c48e5f897\" (UID: \"2d1184f0-d306-4de0-a4f3-016c48e5f897\") "
Oct 30 21:15:20.825069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.824144    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1184f0-d306-4de0-a4f3-016c48e5f897-kube-api-access-467ll\" (UniqueName: \"kubernetes.io/projected/2d1184f0-d306-4de0-a4f3-016c48e5f897-kube-api-access-467ll\") pod \"2d1184f0-d306-4de0-a4f3-016c48e5f897\" (UID: \"2d1184f0-d306-4de0-a4f3-016c48e5f897\") "
Oct 30 21:15:20.825341 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.825316    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1184f0-d306-4de0-a4f3-016c48e5f897-maas-api-service-ca" pod "2d1184f0-d306-4de0-a4f3-016c48e5f897" (UID: "2d1184f0-d306-4de0-a4f3-016c48e5f897"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.827944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.827905    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1184f0-d306-4de0-a4f3-016c48e5f897-kube-api-access-467ll" pod "2d1184f0-d306-4de0-a4f3-016c48e5f897" (UID: "2d1184f0-d306-4de0-a4f3-016c48e5f897"). InnerVolumeSpecName "kube-api-access-467ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.828987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.828937    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"]
Oct 30 21:15:20.834185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.834161    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889915-9kbhj"]
Oct 30 21:15:20.838485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.838461    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"]
Oct 30 21:15:20.868876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.868830    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"
Oct 30 21:15:20.925236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.925210    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe530dfb-4534-49d6-91d3-28f198c77d88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe530dfb-4534-49d6-91d3-28f198c77d88-maas-api-service-ca\") pod \"fe530dfb-4534-49d6-91d3-28f198c77d88\" (UID: \"fe530dfb-4534-49d6-91d3-28f198c77d88\") "
Oct 30 21:15:20.925596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.925575    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe530dfb-4534-49d6-91d3-28f198c77d88-kube-api-access-kt4d5\" (UniqueName: \"kubernetes.io/projected/fe530dfb-4534-49d6-91d3-28f198c77d88-kube-api-access-kt4d5\") pod \"fe530dfb-4534-49d6-91d3-28f198c77d88\" (UID: \"fe530dfb-4534-49d6-91d3-28f198c77d88\") "
Oct 30 21:15:20.926526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.925897    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe530dfb-4534-49d6-91d3-28f198c77d88-maas-api-service-ca" pod "fe530dfb-4534-49d6-91d3-28f198c77d88" (UID: "fe530dfb-4534-49d6-91d3-28f198c77d88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.926765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.926750    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1184f0-d306-4de0-a4f3-016c48e5f897-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.926881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.926867    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-467ll\" (UniqueName: \"kubernetes.io/projected/2d1184f0-d306-4de0-a4f3-016c48e5f897-kube-api-access-467ll\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.928291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:20.928266    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe530dfb-4534-49d6-91d3-28f198c77d88-kube-api-access-kt4d5" pod "fe530dfb-4534-49d6-91d3-28f198c77d88" (UID: "fe530dfb-4534-49d6-91d3-28f198c77d88"). InnerVolumeSpecName "kube-api-access-kt4d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:21.037771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.033413    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4d5\" (UniqueName: \"kubernetes.io/projected/fe530dfb-4534-49d6-91d3-28f198c77d88-kube-api-access-kt4d5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.037771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.033456    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe530dfb-4534-49d6-91d3-28f198c77d88-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.386677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.386363    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"]
Oct 30 21:15:21.388238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388210    4810 cpu_manager.go:414] "containerMap: removing container" podUID="18183221-a049-4171-add0-1810e4939630" containerName="cleanup"
Oct 30 21:15:21.388238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388237    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18183221-a049-4171-add0-1810e4939630" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388254    4810 cpu_manager.go:414] "containerMap: removing container" podUID="69815735-5039-4348-ae5d-68881b8555bd" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388262    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69815735-5039-4348-ae5d-68881b8555bd" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388275    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388286    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388304    4810 cpu_manager.go:414] "containerMap: removing container" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388311    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388326    4810 cpu_manager.go:414] "containerMap: removing container" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388333    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388344    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388351    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388366    4810 cpu_manager.go:414] "containerMap: removing container" podUID="18183221-a049-4171-add0-1810e4939630" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388373    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18183221-a049-4171-add0-1810e4939630" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388388    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388395    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388405    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388411    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388431    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388438    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerName="cleanup"
Oct 30 21:15:21.388441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388454    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388461    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388474    4810 cpu_manager.go:414] "containerMap: removing container" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388480    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388493    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388500    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388517    4810 cpu_manager.go:414] "containerMap: removing container" podUID="69815735-5039-4348-ae5d-68881b8555bd" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388525    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69815735-5039-4348-ae5d-68881b8555bd" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388542    4810 cpu_manager.go:414] "containerMap: removing container" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388549    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388560    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388566    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388579    4810 cpu_manager.go:414] "containerMap: removing container" podUID="69815735-5039-4348-ae5d-68881b8555bd" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388587    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69815735-5039-4348-ae5d-68881b8555bd" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388613    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388620    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388633    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388640    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388657    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388664    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388676    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388683    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388870    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388885    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388898    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388913    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="18183221-a049-4171-add0-1810e4939630" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388921    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="69815735-5039-4348-ae5d-68881b8555bd" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388930    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388940    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388948    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388957    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388966    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="18183221-a049-4171-add0-1810e4939630" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388977    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388986    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.388996    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.389007    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.389018    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="69815735-5039-4348-ae5d-68881b8555bd" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.389026    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.389035    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.389047    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.389058    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.389068    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerName="cleanup"
Oct 30 21:15:21.391500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.389077    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerName="cleanup"
Oct 30 21:15:21.408104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.408078    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"]
Oct 30 21:15:21.409005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.408963    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.439066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.438985    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.439066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.439037    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6919133-5577-44c1-bdb0-565eb839eb6e-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.439491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.439237    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.439491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.439319    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.439491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.439393    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.439491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.439444    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghcpb\" (UniqueName: \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-kube-api-access-ghcpb\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.439735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.439497    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.439735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.439516    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.439735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.439560    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.541555 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.541526    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.541736 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.541582    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.541736 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.541624    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.541736 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.541654    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghcpb\" (UniqueName: \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-kube-api-access-ghcpb\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.541736 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.541691    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.541736 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.541709    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.542089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.541740    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.542089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.541763    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.542089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.541784    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6919133-5577-44c1-bdb0-565eb839eb6e-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.542255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.542111    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.542255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.542219    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.542412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.542388    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.542521 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.542398    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6919133-5577-44c1-bdb0-565eb839eb6e-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.542837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.542811    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.545128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.545102    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.545285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.545263    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.550671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.550649    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.552088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.552070    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghcpb\" (UniqueName: \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-kube-api-access-ghcpb\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.648020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.647987    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66" event={"ID":"fe530dfb-4534-49d6-91d3-28f198c77d88","Type":"ContainerDied","Data":"4357107acfe1dfbf1265b043bc24188165aa77e12da8bfec3776945d29bd1df4"}
Oct 30 21:15:21.648525 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.648010    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"
Oct 30 21:15:21.648618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.648516    4810 scope.go:122] "RemoveContainer" containerID="562560ddebe88225b444f459213c4ca737022e0cd328f3916266b134129524f3"
Oct 30 21:15:21.651094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.651067    4810 generic.go:356] "Generic (PLEG): container finished" podID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerID="667a3ff662778bf5e97861fdf5cd46be5c3faa3d45b39706c35d1380613d1463" exitCode=60
Oct 30 21:15:21.651203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.651169    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" event={"ID":"f91a2f39-f9cc-4bd9-8794-98d575ca8481","Type":"ContainerDied","Data":"667a3ff662778bf5e97861fdf5cd46be5c3faa3d45b39706c35d1380613d1463"}
Oct 30 21:15:21.651557 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.651538    4810 scope.go:122] "RemoveContainer" containerID="667a3ff662778bf5e97861fdf5cd46be5c3faa3d45b39706c35d1380613d1463"
Oct 30 21:15:21.651868 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:21.651834    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q_odh-ai-gateway-infra(f91a2f39-f9cc-4bd9-8794-98d575ca8481)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481"
Oct 30 21:15:21.664391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.664315    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"
Oct 30 21:15:21.670939 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.670285    4810 scope.go:122] "RemoveContainer" containerID="a1119e0608305dab35170f691deaef9a6daa59e114cbf1f6f9cc01b35098f798"
Oct 30 21:15:21.679827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.679706    4810 generic.go:356] "Generic (PLEG): container finished" podID="def80bf3-e036-460e-9543-114a136c81fe" containerID="6cdcafd9e516fdd680b6151972c4a4474a65cc4984835a36aa58ad8edfbdc54d" exitCode=60
Oct 30 21:15:21.684887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.684810    4810 generic.go:356] "Generic (PLEG): container finished" podID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerID="e89fb4ae33c686c5e5dbac8cf2acc88f3da57c313b0df2c9ae68999fea36e381" exitCode=60
Oct 30 21:15:21.688765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.688730    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18183221-a049-4171-add0-1810e4939630" path="/var/lib/kubelet/pods/18183221-a049-4171-add0-1810e4939630/volumes"
Oct 30 21:15:21.689779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.689675    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3328918f-c5a6-4562-acbe-7d0f10ec1329" path="/var/lib/kubelet/pods/3328918f-c5a6-4562-acbe-7d0f10ec1329/volumes"
Oct 30 21:15:21.690518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.690499    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69815735-5039-4348-ae5d-68881b8555bd" path="/var/lib/kubelet/pods/69815735-5039-4348-ae5d-68881b8555bd/volumes"
Oct 30 21:15:21.692189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.692169    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" path="/var/lib/kubelet/pods/80ae6ddf-42ce-46fb-bae7-9facca564fbf/volumes"
Oct 30 21:15:21.693424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.693404    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060" path="/var/lib/kubelet/pods/e9df251a-ccad-45c9-a477-2fa0e8f67060/volumes"
Oct 30 21:15:21.694144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.694111    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" path="/var/lib/kubelet/pods/fb4d6041-3a3c-4980-9f2f-849b4bb7d457/volumes"
Oct 30 21:15:21.695598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.695560    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g" event={"ID":"2d1184f0-d306-4de0-a4f3-016c48e5f897","Type":"ContainerDied","Data":"f6d137d1b838749db4ee25f85f1c5fb9cae33a185b0f1b88fa438c810396d1cf"}
Oct 30 21:15:21.695774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.695754    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts" event={"ID":"def80bf3-e036-460e-9543-114a136c81fe","Type":"ContainerDied","Data":"6cdcafd9e516fdd680b6151972c4a4474a65cc4984835a36aa58ad8edfbdc54d"}
Oct 30 21:15:21.695918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.695901    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" event={"ID":"c29ca7de-8f46-4550-9e3b-58e215775d2d","Type":"ContainerDied","Data":"e89fb4ae33c686c5e5dbac8cf2acc88f3da57c313b0df2c9ae68999fea36e381"}
Oct 30 21:15:21.696563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.696547    4810 scope.go:122] "RemoveContainer" containerID="e89fb4ae33c686c5e5dbac8cf2acc88f3da57c313b0df2c9ae68999fea36e381"
Oct 30 21:15:21.697026 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:21.697005    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24_odh-ai-gateway-infra(c29ca7de-8f46-4550-9e3b-58e215775d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d"
Oct 30 21:15:21.698518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.698496    4810 scope.go:122] "RemoveContainer" containerID="6cdcafd9e516fdd680b6151972c4a4474a65cc4984835a36aa58ad8edfbdc54d"
Oct 30 21:15:21.698887 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:21.698850    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts_odh-ai-gateway-infra(def80bf3-e036-460e-9543-114a136c81fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts" podUID="def80bf3-e036-460e-9543-114a136c81fe"
Oct 30 21:15:21.715014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.714904    4810 scope.go:122] "RemoveContainer" containerID="3014753c5c49209932ecb12060745609c53411b8d2e7c5c7ab759cce527c36ea"
Oct 30 21:15:21.732781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.732757    4810 scope.go:122] "RemoveContainer" containerID="72fe7ba4d4a56a843fffa0e72bc0cd329c5d21d25237f7c8de618382b8efc893"
Oct 30 21:15:21.750843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.750791    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"]
Oct 30 21:15:21.770725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.758053    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-176db5-29889915-rdv66"]
Oct 30 21:15:21.770725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.769265    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:21.779177 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.778833    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"]
Oct 30 21:15:21.782463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.782443    4810 scope.go:122] "RemoveContainer" containerID="d833d7f74526bdb84d58d45063bab735a8fc861c7469b7cd29555de038ee88b8"
Oct 30 21:15:21.785232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.784426    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-345bc4-29889915-g2s5g"]
Oct 30 21:15:21.864905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.863792    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"
Oct 30 21:15:21.950924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.950649    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774fc560-ba2d-4285-8142-be854b22ffc6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/774fc560-ba2d-4285-8142-be854b22ffc6-maas-api-service-ca\") pod \"774fc560-ba2d-4285-8142-be854b22ffc6\" (UID: \"774fc560-ba2d-4285-8142-be854b22ffc6\") "
Oct 30 21:15:21.951097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.950924    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774fc560-ba2d-4285-8142-be854b22ffc6-kube-api-access-swxt4\" (UniqueName: \"kubernetes.io/projected/774fc560-ba2d-4285-8142-be854b22ffc6-kube-api-access-swxt4\") pod \"774fc560-ba2d-4285-8142-be854b22ffc6\" (UID: \"774fc560-ba2d-4285-8142-be854b22ffc6\") "
Oct 30 21:15:21.951385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.951355    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774fc560-ba2d-4285-8142-be854b22ffc6-maas-api-service-ca" pod "774fc560-ba2d-4285-8142-be854b22ffc6" (UID: "774fc560-ba2d-4285-8142-be854b22ffc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:21.951679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.951611    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/774fc560-ba2d-4285-8142-be854b22ffc6-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.954229 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.954197    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774fc560-ba2d-4285-8142-be854b22ffc6-kube-api-access-swxt4" pod "774fc560-ba2d-4285-8142-be854b22ffc6" (UID: "774fc560-ba2d-4285-8142-be854b22ffc6"). InnerVolumeSpecName "kube-api-access-swxt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:21.995273 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:21.995240    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"
Oct 30 21:15:22.053365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.053137    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-kube-api-access-s25pc\" (UniqueName: \"kubernetes.io/projected/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-kube-api-access-s25pc\") pod \"676f8fee-509a-4b6f-8fe0-ceaaae8f709a\" (UID: \"676f8fee-509a-4b6f-8fe0-ceaaae8f709a\") "
Oct 30 21:15:22.053365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.053349    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-maas-api-service-ca\") pod \"676f8fee-509a-4b6f-8fe0-ceaaae8f709a\" (UID: \"676f8fee-509a-4b6f-8fe0-ceaaae8f709a\") "
Oct 30 21:15:22.054203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.054181    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swxt4\" (UniqueName: \"kubernetes.io/projected/774fc560-ba2d-4285-8142-be854b22ffc6-kube-api-access-swxt4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.054583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.054557    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-maas-api-service-ca" pod "676f8fee-509a-4b6f-8fe0-ceaaae8f709a" (UID: "676f8fee-509a-4b6f-8fe0-ceaaae8f709a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:22.081919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.070834    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-kube-api-access-s25pc" pod "676f8fee-509a-4b6f-8fe0-ceaaae8f709a" (UID: "676f8fee-509a-4b6f-8fe0-ceaaae8f709a"). InnerVolumeSpecName "kube-api-access-s25pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:22.081919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.073804    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"
Oct 30 21:15:22.155950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.155920    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"]
Oct 30 21:15:22.157187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.156372    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84174e23-a505-4963-a57e-41762b1faec7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84174e23-a505-4963-a57e-41762b1faec7-maas-api-service-ca\") pod \"84174e23-a505-4963-a57e-41762b1faec7\" (UID: \"84174e23-a505-4963-a57e-41762b1faec7\") "
Oct 30 21:15:22.157187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.156461    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84174e23-a505-4963-a57e-41762b1faec7-kube-api-access-lcn9w\" (UniqueName: \"kubernetes.io/projected/84174e23-a505-4963-a57e-41762b1faec7-kube-api-access-lcn9w\") pod \"84174e23-a505-4963-a57e-41762b1faec7\" (UID: \"84174e23-a505-4963-a57e-41762b1faec7\") "
Oct 30 21:15:22.157187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.156770    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84174e23-a505-4963-a57e-41762b1faec7-maas-api-service-ca" pod "84174e23-a505-4963-a57e-41762b1faec7" (UID: "84174e23-a505-4963-a57e-41762b1faec7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:22.157187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.157135    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.157187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.157153    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84174e23-a505-4963-a57e-41762b1faec7-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.157187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.157165    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s25pc\" (UniqueName: \"kubernetes.io/projected/676f8fee-509a-4b6f-8fe0-ceaaae8f709a-kube-api-access-s25pc\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.159089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.158652    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:22.159089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.158715    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:22.159089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.158744    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:22.159089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.158743    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84174e23-a505-4963-a57e-41762b1faec7-kube-api-access-lcn9w" pod "84174e23-a505-4963-a57e-41762b1faec7" (UID: "84174e23-a505-4963-a57e-41762b1faec7"). InnerVolumeSpecName "kube-api-access-lcn9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:22.258683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.258656    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcn9w\" (UniqueName: \"kubernetes.io/projected/84174e23-a505-4963-a57e-41762b1faec7-kube-api-access-lcn9w\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.699200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.699158    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" event={"ID":"f6919133-5577-44c1-bdb0-565eb839eb6e","Type":"ContainerStarted","Data":"0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6"}
Oct 30 21:15:22.699693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.699220    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" event={"ID":"f6919133-5577-44c1-bdb0-565eb839eb6e","Type":"ContainerStarted","Data":"dd0e438ba327e055437d878e358b2b3f109d3094e55b9521f5bb8c5509758470"}
Oct 30 21:15:22.703810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.703786    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"
Oct 30 21:15:22.703994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.703785    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt" event={"ID":"676f8fee-509a-4b6f-8fe0-ceaaae8f709a","Type":"ContainerDied","Data":"48182dfa7a0e01221774790476bd7793b51e886c1f87fd110e1cc7cfa4f3671c"}
Oct 30 21:15:22.703994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.703929    4810 scope.go:122] "RemoveContainer" containerID="46d2e17fac0f8aed4f5ce39de4a179499a1b7e7ee7fb51d907a2bfda78628e76"
Oct 30 21:15:22.710703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.710678    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"
Oct 30 21:15:22.710826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.710710    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr" event={"ID":"84174e23-a505-4963-a57e-41762b1faec7","Type":"ContainerDied","Data":"e0a34bc047ef3c4896a1624af77bdb79197b73b618a93acf5fe54a22e201d7cb"}
Oct 30 21:15:22.715664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.715637    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9" event={"ID":"774fc560-ba2d-4285-8142-be854b22ffc6","Type":"ContainerDied","Data":"0c03591a58c7fe1105fb6fe2b943a1aa1c0f0cb540c6a90c9948fab4c26d4d86"}
Oct 30 21:15:22.715791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.715665    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"
Oct 30 21:15:22.717839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.717738    4810 scope.go:122] "RemoveContainer" containerID="aab06f268172fe1cca243106f99a989f9f7075ecb1e0dd58b4552cb4efe4904d"
Oct 30 21:15:22.734375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.734303    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" podStartSLOduration=1.734277112 podStartE2EDuration="1.734277112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:22.724139016 +0000 UTC m=+2289.215955586" watchObservedRunningTime="2026-10-30 21:15:22.734277112 +0000 UTC m=+2289.226093657"
Oct 30 21:15:22.742599 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.741543    4810 scope.go:122] "RemoveContainer" containerID="7f822171b73ffe509ed2ba93f814e8570e34d2c71a92f30ffb625f497378acf3"
Oct 30 21:15:22.749660 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.749626    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"]
Oct 30 21:15:22.755888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.755844    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"]
Oct 30 21:15:22.761535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.761511    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f147de-29889915-9h9kt"]
Oct 30 21:15:22.767418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.767395    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"]
Oct 30 21:15:22.775247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.775221    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889915-ctvnr"]
Oct 30 21:15:22.782644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.782619    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:22.792761 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.792700    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"]
Oct 30 21:15:22.800281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.800255    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889915-bm2z9"]
Oct 30 21:15:22.813030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.812929    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"]
Oct 30 21:15:22.855414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.853939    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"]
Oct 30 21:15:22.995983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:22.995952    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"
Oct 30 21:15:23.104359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.078408    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91a2f39-f9cc-4bd9-8794-98d575ca8481-kube-api-access-sbsv2\" (UniqueName: \"kubernetes.io/projected/f91a2f39-f9cc-4bd9-8794-98d575ca8481-kube-api-access-sbsv2\") pod \"f91a2f39-f9cc-4bd9-8794-98d575ca8481\" (UID: \"f91a2f39-f9cc-4bd9-8794-98d575ca8481\") "
Oct 30 21:15:23.104359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.078506    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91a2f39-f9cc-4bd9-8794-98d575ca8481-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91a2f39-f9cc-4bd9-8794-98d575ca8481-maas-api-service-ca\") pod \"f91a2f39-f9cc-4bd9-8794-98d575ca8481\" (UID: \"f91a2f39-f9cc-4bd9-8794-98d575ca8481\") "
Oct 30 21:15:23.104968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.104905    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91a2f39-f9cc-4bd9-8794-98d575ca8481-maas-api-service-ca" pod "f91a2f39-f9cc-4bd9-8794-98d575ca8481" (UID: "f91a2f39-f9cc-4bd9-8794-98d575ca8481"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:23.214937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.188127    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91a2f39-f9cc-4bd9-8794-98d575ca8481-kube-api-access-sbsv2" pod "f91a2f39-f9cc-4bd9-8794-98d575ca8481" (UID: "f91a2f39-f9cc-4bd9-8794-98d575ca8481"). InnerVolumeSpecName "kube-api-access-sbsv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:23.222992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.217591    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91a2f39-f9cc-4bd9-8794-98d575ca8481-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:23.222992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.217628    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbsv2\" (UniqueName: \"kubernetes.io/projected/f91a2f39-f9cc-4bd9-8794-98d575ca8481-kube-api-access-sbsv2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:23.238197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.238172    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"
Oct 30 21:15:23.242440 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.242420    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"
Oct 30 21:15:23.319242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.319203    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29ca7de-8f46-4550-9e3b-58e215775d2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29ca7de-8f46-4550-9e3b-58e215775d2d-maas-api-service-ca\") pod \"c29ca7de-8f46-4550-9e3b-58e215775d2d\" (UID: \"c29ca7de-8f46-4550-9e3b-58e215775d2d\") "
Oct 30 21:15:23.319520 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.319500    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29ca7de-8f46-4550-9e3b-58e215775d2d-kube-api-access-zqc6c\" (UniqueName: \"kubernetes.io/projected/c29ca7de-8f46-4550-9e3b-58e215775d2d-kube-api-access-zqc6c\") pod \"c29ca7de-8f46-4550-9e3b-58e215775d2d\" (UID: \"c29ca7de-8f46-4550-9e3b-58e215775d2d\") "
Oct 30 21:15:23.319724 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.319699    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def80bf3-e036-460e-9543-114a136c81fe-kube-api-access-dtr49\" (UniqueName: \"kubernetes.io/projected/def80bf3-e036-460e-9543-114a136c81fe-kube-api-access-dtr49\") pod \"def80bf3-e036-460e-9543-114a136c81fe\" (UID: \"def80bf3-e036-460e-9543-114a136c81fe\") "
Oct 30 21:15:23.319897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.319879    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def80bf3-e036-460e-9543-114a136c81fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def80bf3-e036-460e-9543-114a136c81fe-maas-api-service-ca\") pod \"def80bf3-e036-460e-9543-114a136c81fe\" (UID: \"def80bf3-e036-460e-9543-114a136c81fe\") "
Oct 30 21:15:23.320945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.320918    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def80bf3-e036-460e-9543-114a136c81fe-maas-api-service-ca" pod "def80bf3-e036-460e-9543-114a136c81fe" (UID: "def80bf3-e036-460e-9543-114a136c81fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:23.321430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.321410    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29ca7de-8f46-4550-9e3b-58e215775d2d-maas-api-service-ca" pod "c29ca7de-8f46-4550-9e3b-58e215775d2d" (UID: "c29ca7de-8f46-4550-9e3b-58e215775d2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:23.324557 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.324537    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29ca7de-8f46-4550-9e3b-58e215775d2d-kube-api-access-zqc6c" pod "c29ca7de-8f46-4550-9e3b-58e215775d2d" (UID: "c29ca7de-8f46-4550-9e3b-58e215775d2d"). InnerVolumeSpecName "kube-api-access-zqc6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:23.324730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.324714    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def80bf3-e036-460e-9543-114a136c81fe-kube-api-access-dtr49" pod "def80bf3-e036-460e-9543-114a136c81fe" (UID: "def80bf3-e036-460e-9543-114a136c81fe"). InnerVolumeSpecName "kube-api-access-dtr49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:23.422028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.421990    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c29ca7de-8f46-4550-9e3b-58e215775d2d-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:23.422028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.422021    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqc6c\" (UniqueName: \"kubernetes.io/projected/c29ca7de-8f46-4550-9e3b-58e215775d2d-kube-api-access-zqc6c\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:23.422028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.422032    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtr49\" (UniqueName: \"kubernetes.io/projected/def80bf3-e036-460e-9543-114a136c81fe-kube-api-access-dtr49\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:23.422257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.422041    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def80bf3-e036-460e-9543-114a136c81fe-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:23.665286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.665253    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897" path="/var/lib/kubelet/pods/2d1184f0-d306-4de0-a4f3-016c48e5f897/volumes"
Oct 30 21:15:23.665868 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.665838    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" path="/var/lib/kubelet/pods/676f8fee-509a-4b6f-8fe0-ceaaae8f709a/volumes"
Oct 30 21:15:23.666322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.666307    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774fc560-ba2d-4285-8142-be854b22ffc6" path="/var/lib/kubelet/pods/774fc560-ba2d-4285-8142-be854b22ffc6/volumes"
Oct 30 21:15:23.667052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.667035    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84174e23-a505-4963-a57e-41762b1faec7" path="/var/lib/kubelet/pods/84174e23-a505-4963-a57e-41762b1faec7/volumes"
Oct 30 21:15:23.667436 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.667421    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88" path="/var/lib/kubelet/pods/fe530dfb-4534-49d6-91d3-28f198c77d88/volumes"
Oct 30 21:15:23.729565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.729540    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"
Oct 30 21:15:23.730231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.730083    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts" event={"ID":"def80bf3-e036-460e-9543-114a136c81fe","Type":"ContainerDied","Data":"1ffcf4a45b9016e14f4c8c88a88b488e565f024f10043202f27229c30b1d42b3"}
Oct 30 21:15:23.730231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.730145    4810 scope.go:122] "RemoveContainer" containerID="6cdcafd9e516fdd680b6151972c4a4474a65cc4984835a36aa58ad8edfbdc54d"
Oct 30 21:15:23.732372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.732271    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24" event={"ID":"c29ca7de-8f46-4550-9e3b-58e215775d2d","Type":"ContainerDied","Data":"260a75dec603afe53c312a32f920af1b24f2a7b810272cab84e85f82adb04205"}
Oct 30 21:15:23.732372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.732291    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"
Oct 30 21:15:23.737616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.737594    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"
Oct 30 21:15:23.737968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.737939    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q" event={"ID":"f91a2f39-f9cc-4bd9-8794-98d575ca8481","Type":"ContainerDied","Data":"559c8da1ee331ef814e178a8265471a5af7794d379639167a91357a08bf33368"}
Oct 30 21:15:23.751227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.751179    4810 scope.go:122] "RemoveContainer" containerID="e89fb4ae33c686c5e5dbac8cf2acc88f3da57c313b0df2c9ae68999fea36e381"
Oct 30 21:15:23.756595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.756574    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"]
Oct 30 21:15:23.766815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.761945    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889915-jlhts"]
Oct 30 21:15:23.766815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.764026    4810 scope.go:122] "RemoveContainer" containerID="667a3ff662778bf5e97861fdf5cd46be5c3faa3d45b39706c35d1380613d1463"
Oct 30 21:15:23.798886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.784995    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:15:23.798886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.785072    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:15:23.798886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.796305    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"]
Oct 30 21:15:23.807889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.803910    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-74f9f4-29889915-vcc24"]
Oct 30 21:15:23.811878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.811280    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"]
Oct 30 21:15:23.818905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:23.818876    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889915-gmb9q"]
Oct 30 21:15:24.783312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:24.783219    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:15:24.783909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:24.783875    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:15:25.157329 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:25.157289    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:15:25.409498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:25.409419    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:25.515194 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:25.515157    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"]
Oct 30 21:15:25.664561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:25.664486    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d" path="/var/lib/kubelet/pods/c29ca7de-8f46-4550-9e3b-58e215775d2d/volumes"
Oct 30 21:15:25.664968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:25.664953    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def80bf3-e036-460e-9543-114a136c81fe" path="/var/lib/kubelet/pods/def80bf3-e036-460e-9543-114a136c81fe/volumes"
Oct 30 21:15:25.665367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:25.665355    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" path="/var/lib/kubelet/pods/f91a2f39-f9cc-4bd9-8794-98d575ca8481/volumes"
Oct 30 21:15:25.796882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:25.795953    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:15:25.796882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:25.796010    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:15:26.784019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:26.783982    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:15:26.784286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:26.784262    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:15:27.791881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:27.789598    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:15:27.791881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:27.789660    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:15:28.365055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:28.365016    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:15:28.789639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:28.789605    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:28.790063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:28.790041    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:28.790842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:28.790818    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:15:30.515473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.515435    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f" podUID="19367ab4-c5b3-4f02-bc95-760f48d1292c" containerName="payload-processing" containerID="cri-o://a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c" gracePeriod=25
Oct 30 21:15:30.835552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.835523    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:15:30.854315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.854254    4810 generic.go:356] "Generic (PLEG): container finished" podID="19367ab4-c5b3-4f02-bc95-760f48d1292c" containerID="a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c" exitCode=0
Oct 30 21:15:30.854526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.854333    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f" event={"ID":"19367ab4-c5b3-4f02-bc95-760f48d1292c","Type":"ContainerDied","Data":"a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c"}
Oct 30 21:15:30.854526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.854393    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f" event={"ID":"19367ab4-c5b3-4f02-bc95-760f48d1292c","Type":"ContainerDied","Data":"6b959d30116bcf89c8ba4fe49be522021ea00beb415b2c8c65f963cd7d662730"}
Oct 30 21:15:30.854526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.854408    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"
Oct 30 21:15:30.854526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.854415    4810 scope.go:122] "RemoveContainer" containerID="a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c"
Oct 30 21:15:30.869941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.869892    4810 scope.go:122] "RemoveContainer" containerID="a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c"
Oct 30 21:15:30.870245 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:30.870223    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c\": container with ID starting with a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c not found: ID does not exist" containerID="a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c"
Oct 30 21:15:30.870314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.870254    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c"} err="failed to get container status \"a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c\": rpc error: code = NotFound desc = could not find container \"a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c\": container with ID starting with a59dc15f9beb855913ec54c0a5ccb1c1a16cb3cfca1429d5464f53d2df23522c not found: ID does not exist"
Oct 30 21:15:30.900763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.900725    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-routing-overlay-volume\") pod \"19367ab4-c5b3-4f02-bc95-760f48d1292c\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") "
Oct 30 21:15:30.900996 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.900839    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-plugins-config-volume\") pod \"19367ab4-c5b3-4f02-bc95-760f48d1292c\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") "
Oct 30 21:15:30.900996 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.900937    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-kube-api-access-8722b\" (UniqueName: \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-kube-api-access-8722b\") pod \"19367ab4-c5b3-4f02-bc95-760f48d1292c\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") "
Oct 30 21:15:30.901128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.901067    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-provider-credentials\") pod \"19367ab4-c5b3-4f02-bc95-760f48d1292c\" (UID: \"19367ab4-c5b3-4f02-bc95-760f48d1292c\") "
Oct 30 21:15:30.901190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.901164    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-routing-overlay-volume" pod "19367ab4-c5b3-4f02-bc95-760f48d1292c" (UID: "19367ab4-c5b3-4f02-bc95-760f48d1292c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:30.901637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.901610    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-plugins-config-volume" pod "19367ab4-c5b3-4f02-bc95-760f48d1292c" (UID: "19367ab4-c5b3-4f02-bc95-760f48d1292c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:30.901829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.901809    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:30.901928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.901835    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19367ab4-c5b3-4f02-bc95-760f48d1292c-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:30.905362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.905324    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-provider-credentials" pod "19367ab4-c5b3-4f02-bc95-760f48d1292c" (UID: "19367ab4-c5b3-4f02-bc95-760f48d1292c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:30.905991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:30.905966    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-kube-api-access-8722b" pod "19367ab4-c5b3-4f02-bc95-760f48d1292c" (UID: "19367ab4-c5b3-4f02-bc95-760f48d1292c"). InnerVolumeSpecName "kube-api-access-8722b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:31.003514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.003477    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8722b\" (UniqueName: \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-kube-api-access-8722b\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:31.003514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.003516    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19367ab4-c5b3-4f02-bc95-760f48d1292c-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:31.181991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.181951    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"]
Oct 30 21:15:31.188793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.188763    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7d6846n4f"]
Oct 30 21:15:31.628637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.628609    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"]
Oct 30 21:15:31.629617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629596    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerName="cleanup"
Oct 30 21:15:31.629677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629617    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4d6041-3a3c-4980-9f2f-849b4bb7d457" containerName="cleanup"
Oct 30 21:15:31.629677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629636    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerName="cleanup"
Oct 30 21:15:31.629677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629641    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerName="cleanup"
Oct 30 21:15:31.629677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629652    4810 cpu_manager.go:414] "containerMap: removing container" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerName="cleanup"
Oct 30 21:15:31.629677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629658    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerName="cleanup"
Oct 30 21:15:31.629677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629666    4810 cpu_manager.go:414] "containerMap: removing container" podUID="84174e23-a505-4963-a57e-41762b1faec7" containerName="cleanup"
Oct 30 21:15:31.629677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629676    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84174e23-a505-4963-a57e-41762b1faec7" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629697    4810 cpu_manager.go:414] "containerMap: removing container" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629702    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629712    4810 cpu_manager.go:414] "containerMap: removing container" podUID="774fc560-ba2d-4285-8142-be854b22ffc6" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629716    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774fc560-ba2d-4285-8142-be854b22ffc6" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629726    4810 cpu_manager.go:414] "containerMap: removing container" podUID="19367ab4-c5b3-4f02-bc95-760f48d1292c" containerName="payload-processing"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629731    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19367ab4-c5b3-4f02-bc95-760f48d1292c" containerName="payload-processing"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629738    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629744    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629756    4810 cpu_manager.go:414] "containerMap: removing container" podUID="def80bf3-e036-460e-9543-114a136c81fe" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629761    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def80bf3-e036-460e-9543-114a136c81fe" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629773    4810 cpu_manager.go:414] "containerMap: removing container" podUID="774fc560-ba2d-4285-8142-be854b22ffc6" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629777    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774fc560-ba2d-4285-8142-be854b22ffc6" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629789    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629796    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629805    4810 cpu_manager.go:414] "containerMap: removing container" podUID="84174e23-a505-4963-a57e-41762b1faec7" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629812    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84174e23-a505-4963-a57e-41762b1faec7" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629819    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629826    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629850    4810 cpu_manager.go:414] "containerMap: removing container" podUID="def80bf3-e036-460e-9543-114a136c81fe" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629868    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def80bf3-e036-460e-9543-114a136c81fe" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629876    4810 cpu_manager.go:414] "containerMap: removing container" podUID="def80bf3-e036-460e-9543-114a136c81fe" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629881    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def80bf3-e036-460e-9543-114a136c81fe" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629889    4810 cpu_manager.go:414] "containerMap: removing container" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629893    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629902    4810 cpu_manager.go:414] "containerMap: removing container" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629909    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629920    4810 cpu_manager.go:414] "containerMap: removing container" podUID="774fc560-ba2d-4285-8142-be854b22ffc6" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629925    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774fc560-ba2d-4285-8142-be854b22ffc6" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629931    4810 cpu_manager.go:414] "containerMap: removing container" podUID="18183221-a049-4171-add0-1810e4939630" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629936    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18183221-a049-4171-add0-1810e4939630" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629943    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629947    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629955    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629960    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe530dfb-4534-49d6-91d3-28f198c77d88" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629968    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerName="cleanup"
Oct 30 21:15:31.629970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.629973    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630097    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="69815735-5039-4348-ae5d-68881b8555bd" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630107    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630116    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="774fc560-ba2d-4285-8142-be854b22ffc6" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630123    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="19367ab4-c5b3-4f02-bc95-760f48d1292c" containerName="payload-processing"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630131    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630138    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="def80bf3-e036-460e-9543-114a136c81fe" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630148    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="774fc560-ba2d-4285-8142-be854b22ffc6" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630154    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9df251a-ccad-45c9-a477-2fa0e8f67060" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630160    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e47096-ee02-476d-ab4c-d7d0846a976a" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630167    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="84174e23-a505-4963-a57e-41762b1faec7" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630172    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630177    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ae6ddf-42ce-46fb-bae7-9facca564fbf" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630186    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="def80bf3-e036-460e-9543-114a136c81fe" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630193    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="def80bf3-e036-460e-9543-114a136c81fe" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630199    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630205    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1184f0-d306-4de0-a4f3-016c48e5f897" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630210    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="18183221-a049-4171-add0-1810e4939630" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630220    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="84174e23-a505-4963-a57e-41762b1faec7" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630227    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630233    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630239    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerName="cleanup"
Oct 30 21:15:31.631146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.630245    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="84174e23-a505-4963-a57e-41762b1faec7" containerName="cleanup"
Oct 30 21:15:31.635399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.635377    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:31.641298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.641274    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:15:31.641464 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.641339    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:31.644543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.644519    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"]
Oct 30 21:15:31.668271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.668243    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19367ab4-c5b3-4f02-bc95-760f48d1292c" path="/var/lib/kubelet/pods/19367ab4-c5b3-4f02-bc95-760f48d1292c/volumes"
Oct 30 21:15:31.714035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.713995    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cbe369a-h4npm\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:31.714213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.714152    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzdb\" (UniqueName: \"kubernetes.io/projected/3eafb34c-4a43-446d-89c8-abe3e77334c2-kube-api-access-clzdb\") pod \"maas-api-revoke-sub-7cbe369a-h4npm\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:31.714379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.714357    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cbe369a-h4npm\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:31.815982 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.815940    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzdb\" (UniqueName: \"kubernetes.io/projected/3eafb34c-4a43-446d-89c8-abe3e77334c2-kube-api-access-clzdb\") pod \"maas-api-revoke-sub-7cbe369a-h4npm\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:31.816196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.816139    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cbe369a-h4npm\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:31.816319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.816303    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cbe369a-h4npm\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:31.817125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.817095    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cbe369a-h4npm\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:31.818650 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.818625    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cbe369a-h4npm\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:31.826292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.826269    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzdb\" (UniqueName: \"kubernetes.io/projected/3eafb34c-4a43-446d-89c8-abe3e77334c2-kube-api-access-clzdb\") pod \"maas-api-revoke-sub-7cbe369a-h4npm\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:31.958217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:31.958119    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:32.262035 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:32.254979    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eafb34c_4a43_446d_89c8_abe3e77334c2.slice/crio-664ac1c2aa8d3973cff9723a428b7b84a29c127ab1f312e36413c75f0764c372 WatchSource:0}: Error finding container 664ac1c2aa8d3973cff9723a428b7b84a29c127ab1f312e36413c75f0764c372: Status 404 returned error can't find the container with id 664ac1c2aa8d3973cff9723a428b7b84a29c127ab1f312e36413c75f0764c372
Oct 30 21:15:32.262035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:32.261109    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"]
Oct 30 21:15:32.883560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:32.883521    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm" event={"ID":"3eafb34c-4a43-446d-89c8-abe3e77334c2","Type":"ContainerStarted","Data":"13d6e89109bdb50f0643357c1e5c881b190b88b1adcf524df09eecc8a4bc2e1c"}
Oct 30 21:15:32.883996 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:32.883592    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm" event={"ID":"3eafb34c-4a43-446d-89c8-abe3e77334c2","Type":"ContainerStarted","Data":"664ac1c2aa8d3973cff9723a428b7b84a29c127ab1f312e36413c75f0764c372"}
Oct 30 21:15:32.907424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:32.907348    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm" podStartSLOduration=1.9073269000000002 podStartE2EDuration="1.9073269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:32.906630792 +0000 UTC m=+2299.398447337" watchObservedRunningTime="2026-10-30 21:15:32.9073269 +0000 UTC m=+2299.399143446"
Oct 30 21:15:33.898023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:33.897985    4810 generic.go:356] "Generic (PLEG): container finished" podID="3eafb34c-4a43-446d-89c8-abe3e77334c2" containerID="13d6e89109bdb50f0643357c1e5c881b190b88b1adcf524df09eecc8a4bc2e1c" exitCode=0
Oct 30 21:15:33.898489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:33.898068    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm" event={"ID":"3eafb34c-4a43-446d-89c8-abe3e77334c2","Type":"ContainerDied","Data":"13d6e89109bdb50f0643357c1e5c881b190b88b1adcf524df09eecc8a4bc2e1c"}
Oct 30 21:15:34.034902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:34.034834    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"]
Oct 30 21:15:34.039159 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:34.039114    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141468aa-l5ns2"]
Oct 30 21:15:35.087102 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.087071    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:35.172732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.172644    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-openshift-ca\") pod \"3eafb34c-4a43-446d-89c8-abe3e77334c2\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") "
Oct 30 21:15:35.172732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.172707    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-xks-ca\") pod \"3eafb34c-4a43-446d-89c8-abe3e77334c2\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") "
Oct 30 21:15:35.173006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.172881    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eafb34c-4a43-446d-89c8-abe3e77334c2-kube-api-access-clzdb\" (UniqueName: \"kubernetes.io/projected/3eafb34c-4a43-446d-89c8-abe3e77334c2-kube-api-access-clzdb\") pod \"3eafb34c-4a43-446d-89c8-abe3e77334c2\" (UID: \"3eafb34c-4a43-446d-89c8-abe3e77334c2\") "
Oct 30 21:15:35.173128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.173091    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-openshift-ca" pod "3eafb34c-4a43-446d-89c8-abe3e77334c2" (UID: "3eafb34c-4a43-446d-89c8-abe3e77334c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:35.173448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.173426    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:35.175136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.175105    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eafb34c-4a43-446d-89c8-abe3e77334c2-kube-api-access-clzdb" pod "3eafb34c-4a43-446d-89c8-abe3e77334c2" (UID: "3eafb34c-4a43-446d-89c8-abe3e77334c2"). InnerVolumeSpecName "kube-api-access-clzdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:35.175378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.175357    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-xks-ca" pod "3eafb34c-4a43-446d-89c8-abe3e77334c2" (UID: "3eafb34c-4a43-446d-89c8-abe3e77334c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:35.274358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.274326    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eafb34c-4a43-446d-89c8-abe3e77334c2-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:35.274358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.274356    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clzdb\" (UniqueName: \"kubernetes.io/projected/3eafb34c-4a43-446d-89c8-abe3e77334c2-kube-api-access-clzdb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:35.664558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.664514    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c869dc88-791f-437c-9b0f-70f46308500c" path="/var/lib/kubelet/pods/c869dc88-791f-437c-9b0f-70f46308500c/volumes"
Oct 30 21:15:35.919346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.919262    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm" event={"ID":"3eafb34c-4a43-446d-89c8-abe3e77334c2","Type":"ContainerDied","Data":"664ac1c2aa8d3973cff9723a428b7b84a29c127ab1f312e36413c75f0764c372"}
Oct 30 21:15:35.919346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.919286    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"
Oct 30 21:15:35.919346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:35.919338    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="664ac1c2aa8d3973cff9723a428b7b84a29c127ab1f312e36413c75f0764c372"
Oct 30 21:15:36.373597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.373511    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"]
Oct 30 21:15:36.374786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.374763    4810 cpu_manager.go:414] "containerMap: removing container" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerName="cleanup"
Oct 30 21:15:36.374850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.374791    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerName="cleanup"
Oct 30 21:15:36.374850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.374823    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerName="cleanup"
Oct 30 21:15:36.374850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.374831    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91a2f39-f9cc-4bd9-8794-98d575ca8481" containerName="cleanup"
Oct 30 21:15:36.374850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.374847    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3eafb34c-4a43-446d-89c8-abe3e77334c2" containerName="revoke-keys"
Oct 30 21:15:36.375014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.374871    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eafb34c-4a43-446d-89c8-abe3e77334c2" containerName="revoke-keys"
Oct 30 21:15:36.375014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.374932    4810 cpu_manager.go:414] "containerMap: removing container" podUID="84174e23-a505-4963-a57e-41762b1faec7" containerName="cleanup"
Oct 30 21:15:36.375014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.374939    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84174e23-a505-4963-a57e-41762b1faec7" containerName="cleanup"
Oct 30 21:15:36.375131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.375119    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eafb34c-4a43-446d-89c8-abe3e77334c2" containerName="revoke-keys"
Oct 30 21:15:36.375168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.375142    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="676f8fee-509a-4b6f-8fe0-ceaaae8f709a" containerName="cleanup"
Oct 30 21:15:36.375168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.375153    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="774fc560-ba2d-4285-8142-be854b22ffc6" containerName="cleanup"
Oct 30 21:15:36.375235 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.375167    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29ca7de-8f46-4550-9e3b-58e215775d2d" containerName="cleanup"
Oct 30 21:15:36.380069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.380047    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.392835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.392804    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"]
Oct 30 21:15:36.495982 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.495941    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp97l\" (UniqueName: \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-kube-api-access-gp97l\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.496196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.496014    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.496196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.496132    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.496295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.496199    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.496342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.496324    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.496421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.496402    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.496475 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.496460    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.496534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.496520    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.496573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.496543    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08995227-51dd-4ba8-9bbe-c4bd9877903e-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.597742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.597698    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.597966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.597761    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.597966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.597802    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.597966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.597900    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.597966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.597929    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08995227-51dd-4ba8-9bbe-c4bd9877903e-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.598217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.598062    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp97l\" (UniqueName: \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-kube-api-access-gp97l\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.598217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.598120    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.598217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.598168    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.598217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.598190    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.598445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.598221    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.598847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.598818    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.598989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.598915    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.599225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.599192    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.599352 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.599263    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08995227-51dd-4ba8-9bbe-c4bd9877903e-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.600419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.600396    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.600739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.600718    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.607346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.607323    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.607893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.607840    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp97l\" (UniqueName: \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-kube-api-access-gp97l\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.701401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.701367    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:36.998325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:36.998293    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"]
Oct 30 21:15:36.998997 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:36.998967    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08995227_51dd_4ba8_9bbe_c4bd9877903e.slice/crio-c977f065fb7c8ce34af7662526fd83b96659eb619eb30adc39a30313a83ba88b WatchSource:0}: Error finding container c977f065fb7c8ce34af7662526fd83b96659eb619eb30adc39a30313a83ba88b: Status 404 returned error can't find the container with id c977f065fb7c8ce34af7662526fd83b96659eb619eb30adc39a30313a83ba88b
Oct 30 21:15:37.003034 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:37.002962    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:37.003121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:37.003083    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:37.003170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:37.003128    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:37.949339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:37.949297    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" event={"ID":"08995227-51dd-4ba8-9bbe-c4bd9877903e","Type":"ContainerStarted","Data":"01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41"}
Oct 30 21:15:37.949744 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:37.949386    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" event={"ID":"08995227-51dd-4ba8-9bbe-c4bd9877903e","Type":"ContainerStarted","Data":"c977f065fb7c8ce34af7662526fd83b96659eb619eb30adc39a30313a83ba88b"}
Oct 30 21:15:37.993880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:37.991171    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" podStartSLOduration=1.991151289 podStartE2EDuration="1.991151289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:37.989227809 +0000 UTC m=+2304.481044356" watchObservedRunningTime="2026-10-30 21:15:37.991151289 +0000 UTC m=+2304.482967835"
Oct 30 21:15:38.702451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:38.702399    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:38.704160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:38.704128    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:15:38.704287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:38.704187    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" podUID="08995227-51dd-4ba8-9bbe-c4bd9877903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:15:39.719882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:39.717278    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:15:39.719882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:39.717338    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" podUID="08995227-51dd-4ba8-9bbe-c4bd9877903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:15:39.889887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:39.888909    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"]
Oct 30 21:15:39.902879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:39.899194    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.161256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.161214    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.161397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.161351    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gll7j\" (UniqueName: \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-kube-api-access-gll7j\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.161466 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.161407    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.161745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.161719    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.164368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.164325    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"]
Oct 30 21:15:40.263405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.263372    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.263578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.263447    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.263578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.263523    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gll7j\" (UniqueName: \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-kube-api-access-gll7j\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.263578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.263560    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.264138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.264116    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.264237 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.264217    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.266244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.266222    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.312878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.312729    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gll7j\" (UniqueName: \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-kube-api-access-gll7j\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.513254 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.513161    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:40.711893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.709767    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:15:40.711893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.709827    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" podUID="08995227-51dd-4ba8-9bbe-c4bd9877903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:15:40.841975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:40.841943    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"]
Oct 30 21:15:40.842680 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:40.842649    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7058cb33_50b0_42ae_a5ba_9ded28af6cc7.slice/crio-909ff9037fc85d2c128a349124e8f822fcea9946db6acb978ca422acfc792865 WatchSource:0}: Error finding container 909ff9037fc85d2c128a349124e8f822fcea9946db6acb978ca422acfc792865: Status 404 returned error can't find the container with id 909ff9037fc85d2c128a349124e8f822fcea9946db6acb978ca422acfc792865
Oct 30 21:15:41.210274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:41.210218    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw" event={"ID":"7058cb33-50b0-42ae-a5ba-9ded28af6cc7","Type":"ContainerStarted","Data":"253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b"}
Oct 30 21:15:41.210467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:41.210298    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw" event={"ID":"7058cb33-50b0-42ae-a5ba-9ded28af6cc7","Type":"ContainerStarted","Data":"909ff9037fc85d2c128a349124e8f822fcea9946db6acb978ca422acfc792865"}
Oct 30 21:15:41.210654 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:41.210636    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:41.232610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:41.232551    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw" podStartSLOduration=2.232537837 podStartE2EDuration="2.232537837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:41.228953389 +0000 UTC m=+2307.720769934" watchObservedRunningTime="2026-10-30 21:15:41.232537837 +0000 UTC m=+2307.724354381"
Oct 30 21:15:41.705397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:41.705347    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:15:41.705626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:41.705424    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" podUID="08995227-51dd-4ba8-9bbe-c4bd9877903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:15:42.502049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.502014    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"]
Oct 30 21:15:42.507492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.507467    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:42.517045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.517022    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:42.517212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.517194    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:15:42.529414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.529380    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"]
Oct 30 21:15:42.643043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.643004    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70718462-229nw\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:42.643043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.643048    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vg5\" (UniqueName: \"kubernetes.io/projected/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-kube-api-access-c4vg5\") pod \"maas-api-revoke-sub-70718462-229nw\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:42.643330 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.643162    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70718462-229nw\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:42.702471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.702426    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:15:42.702659 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.702485    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" podUID="08995227-51dd-4ba8-9bbe-c4bd9877903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:15:42.744330 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.744294    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70718462-229nw\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:42.744512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.744337    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vg5\" (UniqueName: \"kubernetes.io/projected/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-kube-api-access-c4vg5\") pod \"maas-api-revoke-sub-70718462-229nw\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:42.744512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.744404    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70718462-229nw\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:42.745201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.745177    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70718462-229nw\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:42.747121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.747098    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70718462-229nw\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:42.755350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.755281    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vg5\" (UniqueName: \"kubernetes.io/projected/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-kube-api-access-c4vg5\") pod \"maas-api-revoke-sub-70718462-229nw\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:42.823549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:42.823508    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:43.147071 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:43.147029    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ad7868_2e30_4e86_abe5_b6cb67b4f5ab.slice/crio-3a9096dc1c3dffaaef17f7e60716a972d4ba249807fa57fb70a3ad2461d881d7 WatchSource:0}: Error finding container 3a9096dc1c3dffaaef17f7e60716a972d4ba249807fa57fb70a3ad2461d881d7: Status 404 returned error can't find the container with id 3a9096dc1c3dffaaef17f7e60716a972d4ba249807fa57fb70a3ad2461d881d7
Oct 30 21:15:43.148185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:43.148159    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"]
Oct 30 21:15:43.257747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:43.257705    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw" event={"ID":"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab","Type":"ContainerStarted","Data":"3a9096dc1c3dffaaef17f7e60716a972d4ba249807fa57fb70a3ad2461d881d7"}
Oct 30 21:15:43.825879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:43.825758    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:43.826419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:43.826395    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:43.827991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:43.827965    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:15:44.267432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:44.267399    4810 generic.go:356] "Generic (PLEG): container finished" podID="09ad7868-2e30-4e86-abe5-b6cb67b4f5ab" containerID="b55bdc53045775fc58fa32d75d1c903973f7cc891b05ebf174c34c2ab6023b05" exitCode=0
Oct 30 21:15:44.267586 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:44.267482    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw" event={"ID":"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab","Type":"ContainerDied","Data":"b55bdc53045775fc58fa32d75d1c903973f7cc891b05ebf174c34c2ab6023b05"}
Oct 30 21:15:45.463917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:45.463890    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:45.483753 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:45.483730    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-openshift-ca\") pod \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") "
Oct 30 21:15:45.483895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:45.483824    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-kube-api-access-c4vg5\" (UniqueName: \"kubernetes.io/projected/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-kube-api-access-c4vg5\") pod \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") "
Oct 30 21:15:45.483975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:45.483948    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-xks-ca\") pod \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\" (UID: \"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab\") "
Oct 30 21:15:45.485491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:45.485447    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-openshift-ca" pod "09ad7868-2e30-4e86-abe5-b6cb67b4f5ab" (UID: "09ad7868-2e30-4e86-abe5-b6cb67b4f5ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:45.489907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:45.489882    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-kube-api-access-c4vg5" pod "09ad7868-2e30-4e86-abe5-b6cb67b4f5ab" (UID: "09ad7868-2e30-4e86-abe5-b6cb67b4f5ab"). InnerVolumeSpecName "kube-api-access-c4vg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:45.490129 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:45.490112    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-xks-ca" pod "09ad7868-2e30-4e86-abe5-b6cb67b4f5ab" (UID: "09ad7868-2e30-4e86-abe5-b6cb67b4f5ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:45.586303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:45.586224    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:45.586303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:45.586261    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vg5\" (UniqueName: \"kubernetes.io/projected/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-kube-api-access-c4vg5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:45.586303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:45.586277    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:46.288648 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:46.288622    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"
Oct 30 21:15:46.288648 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:46.288637    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw" event={"ID":"09ad7868-2e30-4e86-abe5-b6cb67b4f5ab","Type":"ContainerDied","Data":"3a9096dc1c3dffaaef17f7e60716a972d4ba249807fa57fb70a3ad2461d881d7"}
Oct 30 21:15:46.288840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:46.288690    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a9096dc1c3dffaaef17f7e60716a972d4ba249807fa57fb70a3ad2461d881d7"
Oct 30 21:15:48.342729 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.342692    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"]
Oct 30 21:15:48.343754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.343731    4810 cpu_manager.go:414] "containerMap: removing container" podUID="09ad7868-2e30-4e86-abe5-b6cb67b4f5ab" containerName="revoke-keys"
Oct 30 21:15:48.343754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.343753    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ad7868-2e30-4e86-abe5-b6cb67b4f5ab" containerName="revoke-keys"
Oct 30 21:15:48.344051 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.344033    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ad7868-2e30-4e86-abe5-b6cb67b4f5ab" containerName="revoke-keys"
Oct 30 21:15:48.348100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.348079    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.352729 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.352692    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:48.354370 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.353499    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:15:48.356907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.356883    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"]
Oct 30 21:15:48.445258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.445216    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf773bc2-zt7d5\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.445417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.445287    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf773bc2-zt7d5\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.445458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.445422    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfb2h\" (UniqueName: \"kubernetes.io/projected/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-kube-api-access-qfb2h\") pod \"maas-api-revoke-sub-bf773bc2-zt7d5\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.547394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.547357    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf773bc2-zt7d5\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.547569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.547411    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf773bc2-zt7d5\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.547569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.547460    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfb2h\" (UniqueName: \"kubernetes.io/projected/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-kube-api-access-qfb2h\") pod \"maas-api-revoke-sub-bf773bc2-zt7d5\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.548197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.548177    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf773bc2-zt7d5\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.549929 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.549905    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf773bc2-zt7d5\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.557264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.557239    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfb2h\" (UniqueName: \"kubernetes.io/projected/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-kube-api-access-qfb2h\") pod \"maas-api-revoke-sub-bf773bc2-zt7d5\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.665868 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.665831    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:48.956622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:48.956587    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"]
Oct 30 21:15:48.957260 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:48.957228    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1b4580_be1b_4e7c_b482_bbc89d2abb30.slice/crio-7b84f9d6edc068513fc01e23bb065fba04e3259c494912ba7e95b2e5bc301e75 WatchSource:0}: Error finding container 7b84f9d6edc068513fc01e23bb065fba04e3259c494912ba7e95b2e5bc301e75: Status 404 returned error can't find the container with id 7b84f9d6edc068513fc01e23bb065fba04e3259c494912ba7e95b2e5bc301e75
Oct 30 21:15:49.320107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:49.320020    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5" event={"ID":"fe1b4580-be1b-4e7c-b482-bbc89d2abb30","Type":"ContainerStarted","Data":"7b84f9d6edc068513fc01e23bb065fba04e3259c494912ba7e95b2e5bc301e75"}
Oct 30 21:15:50.329908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:50.329809    4810 generic.go:356] "Generic (PLEG): container finished" podID="fe1b4580-be1b-4e7c-b482-bbc89d2abb30" containerID="c26c641d19b9a065e85b95fa0942e1fce0749c5192969af7940449851cb9c586" exitCode=0
Oct 30 21:15:50.329908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:50.329890    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5" event={"ID":"fe1b4580-be1b-4e7c-b482-bbc89d2abb30","Type":"ContainerDied","Data":"c26c641d19b9a065e85b95fa0942e1fce0749c5192969af7940449851cb9c586"}
Oct 30 21:15:51.546309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:51.545926    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:51.598726 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:51.598632    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-openshift-ca\") pod \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") "
Oct 30 21:15:51.598938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:51.598874    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-xks-ca\") pod \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") "
Oct 30 21:15:51.598938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:51.598918    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-kube-api-access-qfb2h\" (UniqueName: \"kubernetes.io/projected/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-kube-api-access-qfb2h\") pod \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\" (UID: \"fe1b4580-be1b-4e7c-b482-bbc89d2abb30\") "
Oct 30 21:15:51.599086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:51.599049    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-openshift-ca" pod "fe1b4580-be1b-4e7c-b482-bbc89d2abb30" (UID: "fe1b4580-be1b-4e7c-b482-bbc89d2abb30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:51.599598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:51.599563    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:51.601835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:51.601811    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-kube-api-access-qfb2h" pod "fe1b4580-be1b-4e7c-b482-bbc89d2abb30" (UID: "fe1b4580-be1b-4e7c-b482-bbc89d2abb30"). InnerVolumeSpecName "kube-api-access-qfb2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:51.602004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:51.601932    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-xks-ca" pod "fe1b4580-be1b-4e7c-b482-bbc89d2abb30" (UID: "fe1b4580-be1b-4e7c-b482-bbc89d2abb30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:51.701463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:51.701166    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:51.701463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:51.701195    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfb2h\" (UniqueName: \"kubernetes.io/projected/fe1b4580-be1b-4e7c-b482-bbc89d2abb30-kube-api-access-qfb2h\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:52.222750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:52.222721    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:15:52.280764 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:52.280729    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"]
Oct 30 21:15:52.348749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:52.348725    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"
Oct 30 21:15:52.348930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:52.348762    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5" event={"ID":"fe1b4580-be1b-4e7c-b482-bbc89d2abb30","Type":"ContainerDied","Data":"7b84f9d6edc068513fc01e23bb065fba04e3259c494912ba7e95b2e5bc301e75"}
Oct 30 21:15:52.348930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:52.348809    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b84f9d6edc068513fc01e23bb065fba04e3259c494912ba7e95b2e5bc301e75"
Oct 30 21:15:53.647886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.645653    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"]
Oct 30 21:15:53.647886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.647459    4810 cpu_manager.go:414] "containerMap: removing container" podUID="fe1b4580-be1b-4e7c-b482-bbc89d2abb30" containerName="revoke-keys"
Oct 30 21:15:53.647886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.647487    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1b4580-be1b-4e7c-b482-bbc89d2abb30" containerName="revoke-keys"
Oct 30 21:15:53.647886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.647841    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1b4580-be1b-4e7c-b482-bbc89d2abb30" containerName="revoke-keys"
Oct 30 21:15:53.685952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.680067    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.710096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.709955    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"]
Oct 30 21:15:53.749281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.749242    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.749487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.749456    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.749558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.749530    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.749617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.749569    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556f9\" (UniqueName: \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-kube-api-access-556f9\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.850895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.850839    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.851067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.851022    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.851156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.851125    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.851225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.851198    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556f9\" (UniqueName: \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-kube-api-access-556f9\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.851803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.851776    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.852001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.851976    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.853523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.853505    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:53.859950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:53.859928    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-556f9\" (UniqueName: \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-kube-api-access-556f9\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:54.021125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:54.021033    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:54.312825 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:54.310927    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"]
Oct 30 21:15:54.366639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:54.366600    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" event={"ID":"1a0255be-6b58-41d9-9d3e-bf23c833b354","Type":"ContainerStarted","Data":"450c1a32dd8e115a3bc9d83df38c5cb3c032d9412760da77c32f9f121b5db598"}
Oct 30 21:15:54.650284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:54.650249    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"]
Oct 30 21:15:54.655166 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:54.655139    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"]
Oct 30 21:15:55.160120 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.160084    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:15:55.378807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:55.378769    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" event={"ID":"1a0255be-6b58-41d9-9d3e-bf23c833b354","Type":"ContainerStarted","Data":"7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65"}
Oct 30 21:15:55.379046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:55.379026    4810 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" secret="" err="secret \"payload-processing-external-model-e2e--a89b3f58-dockercfg-cssz9\" not found"
Oct 30 21:15:55.379142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:55.379116    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:15:55.409644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:55.409585    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" podStartSLOduration=2.409566405 podStartE2EDuration="2.409566405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:55.403900571 +0000 UTC m=+2321.895717116" watchObservedRunningTime="2026-10-30 21:15:55.409566405 +0000 UTC m=+2321.901382951"
Oct 30 21:15:55.475467 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.475376    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6741b1/agc-cred-cd28d0b544a0: secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:55.475467 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.475402    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8: secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:55.475467 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.475425    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-plugins-e2e-extmod-a-6741b1: configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:15:55.475467 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.475461    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials podName:1a0255be-6b58-41d9-9d3e-bf23c833b354 nodeName:}" failed. No retries permitted until 2026-10-30 21:15:55.975440618 +0000 UTC m=+2322.467257141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354") : secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:55.475799 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.475495    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume podName:1a0255be-6b58-41d9-9d3e-bf23c833b354 nodeName:}" failed. No retries permitted until 2026-10-30 21:15:55.975483813 +0000 UTC m=+2322.467300341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:15:55.983760 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.983698    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-plugins-e2e-extmod-a-6741b1: configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:15:55.984278 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.983804    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume podName:1a0255be-6b58-41d9-9d3e-bf23c833b354 nodeName:}" failed. No retries permitted until 2026-10-30 21:15:56.98378516 +0000 UTC m=+2323.475601683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:15:55.984278 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.983916    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6741b1/agc-cred-cd28d0b544a0: secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:55.984278 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.983933    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8: secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:55.984278 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:55.983981    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials podName:1a0255be-6b58-41d9-9d3e-bf23c833b354 nodeName:}" failed. No retries permitted until 2026-10-30 21:15:56.983966583 +0000 UTC m=+2323.475783121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354") : secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:56.999262 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:56.999224    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-plugins-e2e-extmod-a-6741b1: configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:15:56.999799 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:56.999307    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume podName:1a0255be-6b58-41d9-9d3e-bf23c833b354 nodeName:}" failed. No retries permitted until 2026-10-30 21:15:58.999286431 +0000 UTC m=+2325.491102958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:15:56.999799 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:56.999417    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6741b1/agc-cred-cd28d0b544a0: secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:56.999799 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:56.999439    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8: secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:56.999799 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:56.999475    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials podName:1a0255be-6b58-41d9-9d3e-bf23c833b354 nodeName:}" failed. No retries permitted until 2026-10-30 21:15:58.999465009 +0000 UTC m=+2325.491281531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354") : secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:57.282427 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.282305    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8" podUID="67c8f50b-9264-4eed-8bce-d04d69583004" containerName="payload-processing" containerID="cri-o://abd76e44455d7633797b880f76987d3a5ed6e13657ac345bd5936b82eaaef410" gracePeriod=25
Oct 30 21:15:57.402965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.402932    4810 generic.go:356] "Generic (PLEG): container finished" podID="67c8f50b-9264-4eed-8bce-d04d69583004" containerID="abd76e44455d7633797b880f76987d3a5ed6e13657ac345bd5936b82eaaef410" exitCode=0
Oct 30 21:15:57.403078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.403005    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8" event={"ID":"67c8f50b-9264-4eed-8bce-d04d69583004","Type":"ContainerDied","Data":"abd76e44455d7633797b880f76987d3a5ed6e13657ac345bd5936b82eaaef410"}
Oct 30 21:15:57.579559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.579526    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:57.726012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.725979    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-kube-api-access-g9bmb\" (UniqueName: \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-kube-api-access-g9bmb\") pod \"67c8f50b-9264-4eed-8bce-d04d69583004\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") "
Oct 30 21:15:57.726012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.726024    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-plugins-config-volume\") pod \"67c8f50b-9264-4eed-8bce-d04d69583004\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") "
Oct 30 21:15:57.726267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.726071    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-routing-overlay-volume\") pod \"67c8f50b-9264-4eed-8bce-d04d69583004\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") "
Oct 30 21:15:57.726399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.726377    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-provider-credentials\") pod \"67c8f50b-9264-4eed-8bce-d04d69583004\" (UID: \"67c8f50b-9264-4eed-8bce-d04d69583004\") "
Oct 30 21:15:57.726521 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.726499    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-plugins-config-volume" pod "67c8f50b-9264-4eed-8bce-d04d69583004" (UID: "67c8f50b-9264-4eed-8bce-d04d69583004"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:57.726581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.726508    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-routing-overlay-volume" pod "67c8f50b-9264-4eed-8bce-d04d69583004" (UID: "67c8f50b-9264-4eed-8bce-d04d69583004"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:57.726987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.726970    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:57.727046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.726989    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c8f50b-9264-4eed-8bce-d04d69583004-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:57.728287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.728263    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-kube-api-access-g9bmb" pod "67c8f50b-9264-4eed-8bce-d04d69583004" (UID: "67c8f50b-9264-4eed-8bce-d04d69583004"). InnerVolumeSpecName "kube-api-access-g9bmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:57.728405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.728378    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-provider-credentials" pod "67c8f50b-9264-4eed-8bce-d04d69583004" (UID: "67c8f50b-9264-4eed-8bce-d04d69583004"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:57.828283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.828198    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:57.828283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:57.828226    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9bmb\" (UniqueName: \"kubernetes.io/projected/67c8f50b-9264-4eed-8bce-d04d69583004-kube-api-access-g9bmb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:58.415699 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.415660    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"
Oct 30 21:15:58.415699 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.415680    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8" event={"ID":"67c8f50b-9264-4eed-8bce-d04d69583004","Type":"ContainerDied","Data":"86e649b4d1811b53c59a85f7c2789a17b1f383b6cf489adb0d393015967ef23a"}
Oct 30 21:15:58.416275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.415750    4810 scope.go:122] "RemoveContainer" containerID="abd76e44455d7633797b880f76987d3a5ed6e13657ac345bd5936b82eaaef410"
Oct 30 21:15:58.448003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.447973    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"]
Oct 30 21:15:58.454909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.454883    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-54bft9st8"]
Oct 30 21:15:58.938799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.938762    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"]
Oct 30 21:15:58.939640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.939624    4810 cpu_manager.go:414] "containerMap: removing container" podUID="67c8f50b-9264-4eed-8bce-d04d69583004" containerName="payload-processing"
Oct 30 21:15:58.939688 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.939643    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c8f50b-9264-4eed-8bce-d04d69583004" containerName="payload-processing"
Oct 30 21:15:58.939783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.939774    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c8f50b-9264-4eed-8bce-d04d69583004" containerName="payload-processing"
Oct 30 21:15:58.944759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.944744    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:58.952365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.952341    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:58.952686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.952672    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:15:58.957750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:58.957722    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"]
Oct 30 21:15:59.050445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.050405    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvbf2\" (UniqueName: \"kubernetes.io/projected/dd3b2b0c-5537-4097-bb14-b3fd13729463-kube-api-access-cvbf2\") pod \"maas-api-revoke-sub-18eed34f-zsvsl\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:59.050604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.050522    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18eed34f-zsvsl\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:59.050681 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:59.050660    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6741b1/agc-cred-cd28d0b544a0: secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:59.050717 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:59.050682    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8: secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:59.050756 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:59.050729    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials podName:1a0255be-6b58-41d9-9d3e-bf23c833b354 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:03.050715192 +0000 UTC m=+2329.542531715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354") : secret "agc-cred-cd28d0b544a0" not found
Oct 30 21:15:59.050800 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:59.050759    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-plugins-e2e-extmod-a-6741b1: configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:15:59.050834 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:15:59.050824    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume podName:1a0255be-6b58-41d9-9d3e-bf23c833b354 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:03.050808285 +0000 UTC m=+2329.542624820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-6741b1" not found
Oct 30 21:15:59.050920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.050893    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18eed34f-zsvsl\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:59.152054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.152018    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18eed34f-zsvsl\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:59.152243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.152093    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbf2\" (UniqueName: \"kubernetes.io/projected/dd3b2b0c-5537-4097-bb14-b3fd13729463-kube-api-access-cvbf2\") pod \"maas-api-revoke-sub-18eed34f-zsvsl\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:59.152243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.152164    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18eed34f-zsvsl\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:59.152853 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.152829    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18eed34f-zsvsl\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:59.154661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.154640    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18eed34f-zsvsl\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:59.161597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.161575    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbf2\" (UniqueName: \"kubernetes.io/projected/dd3b2b0c-5537-4097-bb14-b3fd13729463-kube-api-access-cvbf2\") pod \"maas-api-revoke-sub-18eed34f-zsvsl\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:59.261791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.261713    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:15:59.556444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.556418    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"]
Oct 30 21:15:59.556790 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:15:59.556547    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3b2b0c_5537_4097_bb14_b3fd13729463.slice/crio-ee1e0a0f4f4c075fbe8eaba4191aa505389bd8ce12812325b141bca13dc3fac4 WatchSource:0}: Error finding container ee1e0a0f4f4c075fbe8eaba4191aa505389bd8ce12812325b141bca13dc3fac4: Status 404 returned error can't find the container with id ee1e0a0f4f4c075fbe8eaba4191aa505389bd8ce12812325b141bca13dc3fac4
Oct 30 21:15:59.650810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.650772    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2" podUID="ed6efebf-75e1-40c3-a8ff-cc6d033c2100" containerName="payload-processing" containerID="cri-o://95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13" gracePeriod=25
Oct 30 21:15:59.664440 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.664411    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c8f50b-9264-4eed-8bce-d04d69583004" path="/var/lib/kubelet/pods/67c8f50b-9264-4eed-8bce-d04d69583004/volumes"
Oct 30 21:15:59.985882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:15:59.985834    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:16:00.064996 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.064962    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-kube-api-access-zdnhr\" (UniqueName: \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-kube-api-access-zdnhr\") pod \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") "
Oct 30 21:16:00.065176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.065011    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-routing-overlay-volume\") pod \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") "
Oct 30 21:16:00.065224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.065199    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-provider-credentials\") pod \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") "
Oct 30 21:16:00.065277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.065235    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-plugins-config-volume\") pod \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\" (UID: \"ed6efebf-75e1-40c3-a8ff-cc6d033c2100\") "
Oct 30 21:16:00.065562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.065450    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-routing-overlay-volume" pod "ed6efebf-75e1-40c3-a8ff-cc6d033c2100" (UID: "ed6efebf-75e1-40c3-a8ff-cc6d033c2100"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:00.065688 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.065621    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-plugins-config-volume" pod "ed6efebf-75e1-40c3-a8ff-cc6d033c2100" (UID: "ed6efebf-75e1-40c3-a8ff-cc6d033c2100"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:00.066015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.065996    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:00.066015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.066015    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:00.067418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.067375    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-kube-api-access-zdnhr" pod "ed6efebf-75e1-40c3-a8ff-cc6d033c2100" (UID: "ed6efebf-75e1-40c3-a8ff-cc6d033c2100"). InnerVolumeSpecName "kube-api-access-zdnhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:00.067759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.067735    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-provider-credentials" pod "ed6efebf-75e1-40c3-a8ff-cc6d033c2100" (UID: "ed6efebf-75e1-40c3-a8ff-cc6d033c2100"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:00.167474 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.167388    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:00.167474 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.167412    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdnhr\" (UniqueName: \"kubernetes.io/projected/ed6efebf-75e1-40c3-a8ff-cc6d033c2100-kube-api-access-zdnhr\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:00.452272 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.452173    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl" event={"ID":"dd3b2b0c-5537-4097-bb14-b3fd13729463","Type":"ContainerStarted","Data":"fc8663455d2b5dce4df22c373b0922fc2270b8c7453e605e058ba7b30f079110"}
Oct 30 21:16:00.452437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.452321    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl" event={"ID":"dd3b2b0c-5537-4097-bb14-b3fd13729463","Type":"ContainerStarted","Data":"ee1e0a0f4f4c075fbe8eaba4191aa505389bd8ce12812325b141bca13dc3fac4"}
Oct 30 21:16:00.453997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.453965    4810 generic.go:356] "Generic (PLEG): container finished" podID="ed6efebf-75e1-40c3-a8ff-cc6d033c2100" containerID="95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13" exitCode=0
Oct 30 21:16:00.454159 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.454000    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2" event={"ID":"ed6efebf-75e1-40c3-a8ff-cc6d033c2100","Type":"ContainerDied","Data":"95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13"}
Oct 30 21:16:00.454159 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.454024    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"
Oct 30 21:16:00.454159 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.454054    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2" event={"ID":"ed6efebf-75e1-40c3-a8ff-cc6d033c2100","Type":"ContainerDied","Data":"a7501e8a8202e613180a8d35d9939cbc07243e1c8126c10f43b768d1b89fd05e"}
Oct 30 21:16:00.454159 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.454083    4810 scope.go:122] "RemoveContainer" containerID="95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13"
Oct 30 21:16:00.466100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.466075    4810 scope.go:122] "RemoveContainer" containerID="95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13"
Oct 30 21:16:00.466377 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:00.466353    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13\": container with ID starting with 95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13 not found: ID does not exist" containerID="95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13"
Oct 30 21:16:00.466423 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.466390    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13"} err="failed to get container status \"95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13\": rpc error: code = NotFound desc = could not find container \"95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13\": container with ID starting with 95b48a33eb10a0976b79e46bf1d86078b8f7a409bf2bdbab8a71073d9e235a13 not found: ID does not exist"
Oct 30 21:16:00.488561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.488511    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl" podStartSLOduration=2.488498382 podStartE2EDuration="2.488498382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:00.4741954 +0000 UTC m=+2326.966011946" watchObservedRunningTime="2026-10-30 21:16:00.488498382 +0000 UTC m=+2326.980314929"
Oct 30 21:16:00.491183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.491162    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"]
Oct 30 21:16:00.503710 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:00.503687    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5978g2nc2"]
Oct 30 21:16:01.394616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:01.390941    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" podUID="1a0255be-6b58-41d9-9d3e-bf23c833b354" containerName="payload-processing" containerID="cri-o://7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65" gracePeriod=25
Oct 30 21:16:01.496177 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:01.496133    4810 generic.go:356] "Generic (PLEG): container finished" podID="dd3b2b0c-5537-4097-bb14-b3fd13729463" containerID="fc8663455d2b5dce4df22c373b0922fc2270b8c7453e605e058ba7b30f079110" exitCode=0
Oct 30 21:16:01.496375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:01.496207    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl" event={"ID":"dd3b2b0c-5537-4097-bb14-b3fd13729463","Type":"ContainerDied","Data":"fc8663455d2b5dce4df22c373b0922fc2270b8c7453e605e058ba7b30f079110"}
Oct 30 21:16:01.879426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:01.879386    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6efebf-75e1-40c3-a8ff-cc6d033c2100" path="/var/lib/kubelet/pods/ed6efebf-75e1-40c3-a8ff-cc6d033c2100/volumes"
Oct 30 21:16:01.961763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:01.961735    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:16:02.032918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.032787    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-kube-api-access-556f9\" (UniqueName: \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-kube-api-access-556f9\") pod \"1a0255be-6b58-41d9-9d3e-bf23c833b354\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") "
Oct 30 21:16:02.033324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.032928    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume\") pod \"1a0255be-6b58-41d9-9d3e-bf23c833b354\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") "
Oct 30 21:16:02.033324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.033002    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials\") pod \"1a0255be-6b58-41d9-9d3e-bf23c833b354\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") "
Oct 30 21:16:02.033324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.033073    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-routing-overlay-volume\") pod \"1a0255be-6b58-41d9-9d3e-bf23c833b354\" (UID: \"1a0255be-6b58-41d9-9d3e-bf23c833b354\") "
Oct 30 21:16:02.033692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.033411    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume" pod "1a0255be-6b58-41d9-9d3e-bf23c833b354" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:02.033820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.033779    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:02.033820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.033782    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-routing-overlay-volume" pod "1a0255be-6b58-41d9-9d3e-bf23c833b354" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:02.035827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.035799    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-kube-api-access-556f9" pod "1a0255be-6b58-41d9-9d3e-bf23c833b354" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354"). InnerVolumeSpecName "kube-api-access-556f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:02.037432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.037404    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials" pod "1a0255be-6b58-41d9-9d3e-bf23c833b354" (UID: "1a0255be-6b58-41d9-9d3e-bf23c833b354"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:02.134979 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.134945    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-556f9\" (UniqueName: \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-kube-api-access-556f9\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:02.134979 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.134975    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a0255be-6b58-41d9-9d3e-bf23c833b354-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:02.135283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.134990    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a0255be-6b58-41d9-9d3e-bf23c833b354-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:02.508548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.508510    4810 generic.go:356] "Generic (PLEG): container finished" podID="1a0255be-6b58-41d9-9d3e-bf23c833b354" containerID="7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65" exitCode=0
Oct 30 21:16:02.509103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.508559    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"
Oct 30 21:16:02.509103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.508591    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" event={"ID":"1a0255be-6b58-41d9-9d3e-bf23c833b354","Type":"ContainerDied","Data":"7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65"}
Oct 30 21:16:02.509103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.508651    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8" event={"ID":"1a0255be-6b58-41d9-9d3e-bf23c833b354","Type":"ContainerDied","Data":"450c1a32dd8e115a3bc9d83df38c5cb3c032d9412760da77c32f9f121b5db598"}
Oct 30 21:16:02.509103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.508673    4810 scope.go:122] "RemoveContainer" containerID="7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65"
Oct 30 21:16:02.523383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.523356    4810 scope.go:122] "RemoveContainer" containerID="7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65"
Oct 30 21:16:02.523717 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:02.523686    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65\": container with ID starting with 7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65 not found: ID does not exist" containerID="7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65"
Oct 30 21:16:02.523792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.523726    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65"} err="failed to get container status \"7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65\": rpc error: code = NotFound desc = could not find container \"7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65\": container with ID starting with 7fe9f452d774c06510c377053232ebbdda15125d5135dda9a80fd8b683212d65 not found: ID does not exist"
Oct 30 21:16:02.537324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.537289    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"]
Oct 30 21:16:02.541909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.541799    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-5799lddh8"]
Oct 30 21:16:02.695161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.695134    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:16:02.743243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.743202    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3b2b0c-5537-4097-bb14-b3fd13729463-kube-api-access-cvbf2\" (UniqueName: \"kubernetes.io/projected/dd3b2b0c-5537-4097-bb14-b3fd13729463-kube-api-access-cvbf2\") pod \"dd3b2b0c-5537-4097-bb14-b3fd13729463\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") "
Oct 30 21:16:02.743445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.743283    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-openshift-ca\") pod \"dd3b2b0c-5537-4097-bb14-b3fd13729463\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") "
Oct 30 21:16:02.743698 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.743667    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-openshift-ca" pod "dd3b2b0c-5537-4097-bb14-b3fd13729463" (UID: "dd3b2b0c-5537-4097-bb14-b3fd13729463"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:02.743924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.743903    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-xks-ca\") pod \"dd3b2b0c-5537-4097-bb14-b3fd13729463\" (UID: \"dd3b2b0c-5537-4097-bb14-b3fd13729463\") "
Oct 30 21:16:02.744717 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.744690    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:02.746804 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.746766    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3b2b0c-5537-4097-bb14-b3fd13729463-kube-api-access-cvbf2" pod "dd3b2b0c-5537-4097-bb14-b3fd13729463" (UID: "dd3b2b0c-5537-4097-bb14-b3fd13729463"). InnerVolumeSpecName "kube-api-access-cvbf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:02.747775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.747722    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-xks-ca" pod "dd3b2b0c-5537-4097-bb14-b3fd13729463" (UID: "dd3b2b0c-5537-4097-bb14-b3fd13729463"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:02.846492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.846455    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvbf2\" (UniqueName: \"kubernetes.io/projected/dd3b2b0c-5537-4097-bb14-b3fd13729463-kube-api-access-cvbf2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:02.846492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:02.846490    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3b2b0c-5537-4097-bb14-b3fd13729463-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:03.136852 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.136797    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"]
Oct 30 21:16:03.138194 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.138164    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ed6efebf-75e1-40c3-a8ff-cc6d033c2100" containerName="payload-processing"
Oct 30 21:16:03.138293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.138200    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6efebf-75e1-40c3-a8ff-cc6d033c2100" containerName="payload-processing"
Oct 30 21:16:03.138293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.138219    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1a0255be-6b58-41d9-9d3e-bf23c833b354" containerName="payload-processing"
Oct 30 21:16:03.138293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.138229    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0255be-6b58-41d9-9d3e-bf23c833b354" containerName="payload-processing"
Oct 30 21:16:03.138293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.138283    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dd3b2b0c-5537-4097-bb14-b3fd13729463" containerName="revoke-keys"
Oct 30 21:16:03.138431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.138295    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3b2b0c-5537-4097-bb14-b3fd13729463" containerName="revoke-keys"
Oct 30 21:16:03.138570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.138557    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3b2b0c-5537-4097-bb14-b3fd13729463" containerName="revoke-keys"
Oct 30 21:16:03.138611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.138590    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6efebf-75e1-40c3-a8ff-cc6d033c2100" containerName="payload-processing"
Oct 30 21:16:03.138611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.138604    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0255be-6b58-41d9-9d3e-bf23c833b354" containerName="payload-processing"
Oct 30 21:16:03.143790 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.143762    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.153550 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.153516    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"]
Oct 30 21:16:03.251849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.251801    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwck\" (UniqueName: \"kubernetes.io/projected/61031fdc-1d64-42e2-bb34-69234ff4b0fe-kube-api-access-7pwck\") pod \"maas-api-revoke-sub-c218e591-h8pk7\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.252052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.252002    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c218e591-h8pk7\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.252285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.252265    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c218e591-h8pk7\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.353768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.353716    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c218e591-h8pk7\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.354026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.353920    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwck\" (UniqueName: \"kubernetes.io/projected/61031fdc-1d64-42e2-bb34-69234ff4b0fe-kube-api-access-7pwck\") pod \"maas-api-revoke-sub-c218e591-h8pk7\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.354104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.354036    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c218e591-h8pk7\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.354538 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.354516    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c218e591-h8pk7\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.356499 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.356471    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c218e591-h8pk7\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.365327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.365303    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwck\" (UniqueName: \"kubernetes.io/projected/61031fdc-1d64-42e2-bb34-69234ff4b0fe-kube-api-access-7pwck\") pod \"maas-api-revoke-sub-c218e591-h8pk7\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.465320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.465231    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:03.545197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.545153    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl" event={"ID":"dd3b2b0c-5537-4097-bb14-b3fd13729463","Type":"ContainerDied","Data":"ee1e0a0f4f4c075fbe8eaba4191aa505389bd8ce12812325b141bca13dc3fac4"}
Oct 30 21:16:03.545565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.545259    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee1e0a0f4f4c075fbe8eaba4191aa505389bd8ce12812325b141bca13dc3fac4"
Oct 30 21:16:03.545565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.545354    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"
Oct 30 21:16:03.664016 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.663981    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0255be-6b58-41d9-9d3e-bf23c833b354" path="/var/lib/kubelet/pods/1a0255be-6b58-41d9-9d3e-bf23c833b354/volumes"
Oct 30 21:16:03.762753 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:03.762719    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61031fdc_1d64_42e2_bb34_69234ff4b0fe.slice/crio-c719e27c76c7a39756d8612e2c6f76c2f8f92a992a48cabfa25f92ddd9dfbd33 WatchSource:0}: Error finding container c719e27c76c7a39756d8612e2c6f76c2f8f92a992a48cabfa25f92ddd9dfbd33: Status 404 returned error can't find the container with id c719e27c76c7a39756d8612e2c6f76c2f8f92a992a48cabfa25f92ddd9dfbd33
Oct 30 21:16:03.763297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:03.763201    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"]
Oct 30 21:16:04.555927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.555801    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7" event={"ID":"61031fdc-1d64-42e2-bb34-69234ff4b0fe","Type":"ContainerStarted","Data":"08041c791c3816fdacacf6db174f6905fac2e76c70ca8cfb4f60b87949917cbd"}
Oct 30 21:16:04.555927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.555869    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7" event={"ID":"61031fdc-1d64-42e2-bb34-69234ff4b0fe","Type":"ContainerStarted","Data":"c719e27c76c7a39756d8612e2c6f76c2f8f92a992a48cabfa25f92ddd9dfbd33"}
Oct 30 21:16:04.578388 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.578325    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7" podStartSLOduration=1.578309651 podStartE2EDuration="1.578309651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:04.572310874 +0000 UTC m=+2331.064127420" watchObservedRunningTime="2026-10-30 21:16:04.578309651 +0000 UTC m=+2331.070126238"
Oct 30 21:16:04.628184 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.628149    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"]
Oct 30 21:16:04.633737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.633710    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:04.643902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.643823    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"]
Oct 30 21:16:04.777136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.777090    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-781a71ad-67rxk\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:04.777848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.777495    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxd7\" (UniqueName: \"kubernetes.io/projected/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-kube-api-access-5kxd7\") pod \"maas-api-revoke-sub-781a71ad-67rxk\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:04.777848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.777674    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-781a71ad-67rxk\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:04.879144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.879105    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-781a71ad-67rxk\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:04.879357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.879203    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-781a71ad-67rxk\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:04.879357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.879316    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxd7\" (UniqueName: \"kubernetes.io/projected/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-kube-api-access-5kxd7\") pod \"maas-api-revoke-sub-781a71ad-67rxk\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:04.879951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.879929    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-781a71ad-67rxk\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:04.881768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.881749    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-781a71ad-67rxk\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:04.889203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.889177    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxd7\" (UniqueName: \"kubernetes.io/projected/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-kube-api-access-5kxd7\") pod \"maas-api-revoke-sub-781a71ad-67rxk\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:04.956697 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:04.956663    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:05.053556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.053525    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"]
Oct 30 21:16:05.062784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.062753    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.074937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.074820    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"]
Oct 30 21:16:05.184167 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.184127    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50db59f0-6q7zt\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.184357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.184197    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50db59f0-6q7zt\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.184413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.184354    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2gv\" (UniqueName: \"kubernetes.io/projected/441766c7-d7c8-4112-99d1-67ff7d064957-kube-api-access-9r2gv\") pod \"maas-api-revoke-sub-50db59f0-6q7zt\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.282071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.273714    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"]
Oct 30 21:16:05.288995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.288958    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2gv\" (UniqueName: \"kubernetes.io/projected/441766c7-d7c8-4112-99d1-67ff7d064957-kube-api-access-9r2gv\") pod \"maas-api-revoke-sub-50db59f0-6q7zt\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.289297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.289279    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50db59f0-6q7zt\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.289349 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.289339    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50db59f0-6q7zt\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.290164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.290144    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50db59f0-6q7zt\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.298905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.293643    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50db59f0-6q7zt\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.299946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.299920    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2gv\" (UniqueName: \"kubernetes.io/projected/441766c7-d7c8-4112-99d1-67ff7d064957-kube-api-access-9r2gv\") pod \"maas-api-revoke-sub-50db59f0-6q7zt\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.392014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.391984    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:05.567689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.567651    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk" event={"ID":"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7","Type":"ContainerStarted","Data":"82d030c08e308fef16625cdcc3e2b42ccc45a846acaff92a9bb4a45d73c6d4e3"}
Oct 30 21:16:05.569642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.569614    4810 generic.go:356] "Generic (PLEG): container finished" podID="61031fdc-1d64-42e2-bb34-69234ff4b0fe" containerID="08041c791c3816fdacacf6db174f6905fac2e76c70ca8cfb4f60b87949917cbd" exitCode=0
Oct 30 21:16:05.569768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.569697    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7" event={"ID":"61031fdc-1d64-42e2-bb34-69234ff4b0fe","Type":"ContainerDied","Data":"08041c791c3816fdacacf6db174f6905fac2e76c70ca8cfb4f60b87949917cbd"}
Oct 30 21:16:05.681533 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:05.681504    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441766c7_d7c8_4112_99d1_67ff7d064957.slice/crio-5fd60491d95c1db49e1242a0431a490c004e048dc8a1d544b8db7708b01ae81e WatchSource:0}: Error finding container 5fd60491d95c1db49e1242a0431a490c004e048dc8a1d544b8db7708b01ae81e: Status 404 returned error can't find the container with id 5fd60491d95c1db49e1242a0431a490c004e048dc8a1d544b8db7708b01ae81e
Oct 30 21:16:05.684732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:05.684707    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"]
Oct 30 21:16:06.591078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.590533    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt" event={"ID":"441766c7-d7c8-4112-99d1-67ff7d064957","Type":"ContainerStarted","Data":"cdaa41483bbb6990ee348932268320f43aa960154e23018b400897b2565198b4"}
Oct 30 21:16:06.591078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.590657    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt" event={"ID":"441766c7-d7c8-4112-99d1-67ff7d064957","Type":"ContainerStarted","Data":"5fd60491d95c1db49e1242a0431a490c004e048dc8a1d544b8db7708b01ae81e"}
Oct 30 21:16:06.593143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.593111    4810 generic.go:356] "Generic (PLEG): container finished" podID="7a13c6af-bfa7-4c2d-a78f-990ca7f451d7" containerID="3001e0ddebd6c5dd05257113ce8840c74de9f2fc1e97bf9f23eb7e6d015cab99" exitCode=0
Oct 30 21:16:06.593287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.593160    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk" event={"ID":"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7","Type":"ContainerDied","Data":"3001e0ddebd6c5dd05257113ce8840c74de9f2fc1e97bf9f23eb7e6d015cab99"}
Oct 30 21:16:06.614341 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.614294    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt" podStartSLOduration=1.614281352 podStartE2EDuration="1.614281352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:06.612254505 +0000 UTC m=+2333.104071054" watchObservedRunningTime="2026-10-30 21:16:06.614281352 +0000 UTC m=+2333.106097896"
Oct 30 21:16:06.787577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.787553    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:06.912983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.912114    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-openshift-ca\") pod \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") "
Oct 30 21:16:06.912983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.912189    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61031fdc-1d64-42e2-bb34-69234ff4b0fe-kube-api-access-7pwck\" (UniqueName: \"kubernetes.io/projected/61031fdc-1d64-42e2-bb34-69234ff4b0fe-kube-api-access-7pwck\") pod \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") "
Oct 30 21:16:06.912983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.912497    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-xks-ca\") pod \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\" (UID: \"61031fdc-1d64-42e2-bb34-69234ff4b0fe\") "
Oct 30 21:16:06.916174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.916120    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-openshift-ca" pod "61031fdc-1d64-42e2-bb34-69234ff4b0fe" (UID: "61031fdc-1d64-42e2-bb34-69234ff4b0fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:06.933195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.933156    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-xks-ca" pod "61031fdc-1d64-42e2-bb34-69234ff4b0fe" (UID: "61031fdc-1d64-42e2-bb34-69234ff4b0fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:06.937834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:06.937796    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61031fdc-1d64-42e2-bb34-69234ff4b0fe-kube-api-access-7pwck" pod "61031fdc-1d64-42e2-bb34-69234ff4b0fe" (UID: "61031fdc-1d64-42e2-bb34-69234ff4b0fe"). InnerVolumeSpecName "kube-api-access-7pwck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:07.014950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.014915    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.014950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.014946    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pwck\" (UniqueName: \"kubernetes.io/projected/61031fdc-1d64-42e2-bb34-69234ff4b0fe-kube-api-access-7pwck\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.014950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.014957    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61031fdc-1d64-42e2-bb34-69234ff4b0fe-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.604913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.604851    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"
Oct 30 21:16:07.605401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.604882    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7" event={"ID":"61031fdc-1d64-42e2-bb34-69234ff4b0fe","Type":"ContainerDied","Data":"c719e27c76c7a39756d8612e2c6f76c2f8f92a992a48cabfa25f92ddd9dfbd33"}
Oct 30 21:16:07.605401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.605015    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c719e27c76c7a39756d8612e2c6f76c2f8f92a992a48cabfa25f92ddd9dfbd33"
Oct 30 21:16:07.607482 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.607170    4810 generic.go:356] "Generic (PLEG): container finished" podID="441766c7-d7c8-4112-99d1-67ff7d064957" containerID="cdaa41483bbb6990ee348932268320f43aa960154e23018b400897b2565198b4" exitCode=0
Oct 30 21:16:07.607482 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.607205    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt" event={"ID":"441766c7-d7c8-4112-99d1-67ff7d064957","Type":"ContainerDied","Data":"cdaa41483bbb6990ee348932268320f43aa960154e23018b400897b2565198b4"}
Oct 30 21:16:07.858561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.858504    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:07.926071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.926036    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-xks-ca\") pod \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") "
Oct 30 21:16:07.926244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.926095    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-kube-api-access-5kxd7\" (UniqueName: \"kubernetes.io/projected/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-kube-api-access-5kxd7\") pod \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") "
Oct 30 21:16:07.926328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.926264    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-openshift-ca\") pod \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\" (UID: \"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7\") "
Oct 30 21:16:07.926744 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.926708    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-openshift-ca" pod "7a13c6af-bfa7-4c2d-a78f-990ca7f451d7" (UID: "7a13c6af-bfa7-4c2d-a78f-990ca7f451d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:07.927178 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.927155    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.928925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.928898    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-xks-ca" pod "7a13c6af-bfa7-4c2d-a78f-990ca7f451d7" (UID: "7a13c6af-bfa7-4c2d-a78f-990ca7f451d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:07.929255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:07.929231    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-kube-api-access-5kxd7" pod "7a13c6af-bfa7-4c2d-a78f-990ca7f451d7" (UID: "7a13c6af-bfa7-4c2d-a78f-990ca7f451d7"). InnerVolumeSpecName "kube-api-access-5kxd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:08.028816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.028775    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.028816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.028804    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxd7\" (UniqueName: \"kubernetes.io/projected/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7-kube-api-access-5kxd7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.636126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.636085    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk" event={"ID":"7a13c6af-bfa7-4c2d-a78f-990ca7f451d7","Type":"ContainerDied","Data":"82d030c08e308fef16625cdcc3e2b42ccc45a846acaff92a9bb4a45d73c6d4e3"}
Oct 30 21:16:08.636126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.636113    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"
Oct 30 21:16:08.636700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.636163    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82d030c08e308fef16625cdcc3e2b42ccc45a846acaff92a9bb4a45d73c6d4e3"
Oct 30 21:16:08.813147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.813121    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:08.840744 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.840706    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-xks-ca\") pod \"441766c7-d7c8-4112-99d1-67ff7d064957\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") "
Oct 30 21:16:08.840936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.840839    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-openshift-ca\") pod \"441766c7-d7c8-4112-99d1-67ff7d064957\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") "
Oct 30 21:16:08.840936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.840911    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441766c7-d7c8-4112-99d1-67ff7d064957-kube-api-access-9r2gv\" (UniqueName: \"kubernetes.io/projected/441766c7-d7c8-4112-99d1-67ff7d064957-kube-api-access-9r2gv\") pod \"441766c7-d7c8-4112-99d1-67ff7d064957\" (UID: \"441766c7-d7c8-4112-99d1-67ff7d064957\") "
Oct 30 21:16:08.841195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.841171    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-openshift-ca" pod "441766c7-d7c8-4112-99d1-67ff7d064957" (UID: "441766c7-d7c8-4112-99d1-67ff7d064957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:08.841619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.841591    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.842955 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.842927    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-xks-ca" pod "441766c7-d7c8-4112-99d1-67ff7d064957" (UID: "441766c7-d7c8-4112-99d1-67ff7d064957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:08.843224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.843193    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441766c7-d7c8-4112-99d1-67ff7d064957-kube-api-access-9r2gv" pod "441766c7-d7c8-4112-99d1-67ff7d064957" (UID: "441766c7-d7c8-4112-99d1-67ff7d064957"). InnerVolumeSpecName "kube-api-access-9r2gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:08.942786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.942708    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441766c7-d7c8-4112-99d1-67ff7d064957-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.942786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:08.942739    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r2gv\" (UniqueName: \"kubernetes.io/projected/441766c7-d7c8-4112-99d1-67ff7d064957-kube-api-access-9r2gv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:09.646063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:09.646031    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"
Oct 30 21:16:09.646473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:09.646055    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt" event={"ID":"441766c7-d7c8-4112-99d1-67ff7d064957","Type":"ContainerDied","Data":"5fd60491d95c1db49e1242a0431a490c004e048dc8a1d544b8db7708b01ae81e"}
Oct 30 21:16:09.646473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:09.646104    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fd60491d95c1db49e1242a0431a490c004e048dc8a1d544b8db7708b01ae81e"
Oct 30 21:16:11.258591 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.258551    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"]
Oct 30 21:16:11.259829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.259805    4810 cpu_manager.go:414] "containerMap: removing container" podUID="61031fdc-1d64-42e2-bb34-69234ff4b0fe" containerName="revoke-keys"
Oct 30 21:16:11.259935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.259832    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61031fdc-1d64-42e2-bb34-69234ff4b0fe" containerName="revoke-keys"
Oct 30 21:16:11.259935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.259885    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7a13c6af-bfa7-4c2d-a78f-990ca7f451d7" containerName="revoke-keys"
Oct 30 21:16:11.259935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.259891    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a13c6af-bfa7-4c2d-a78f-990ca7f451d7" containerName="revoke-keys"
Oct 30 21:16:11.259935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.259901    4810 cpu_manager.go:414] "containerMap: removing container" podUID="441766c7-d7c8-4112-99d1-67ff7d064957" containerName="revoke-keys"
Oct 30 21:16:11.259935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.259907    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441766c7-d7c8-4112-99d1-67ff7d064957" containerName="revoke-keys"
Oct 30 21:16:11.260103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.260060    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a13c6af-bfa7-4c2d-a78f-990ca7f451d7" containerName="revoke-keys"
Oct 30 21:16:11.260103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.260069    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="441766c7-d7c8-4112-99d1-67ff7d064957" containerName="revoke-keys"
Oct 30 21:16:11.260103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.260083    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="61031fdc-1d64-42e2-bb34-69234ff4b0fe" containerName="revoke-keys"
Oct 30 21:16:11.264043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.264024    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.275981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.275949    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"]
Oct 30 21:16:11.279653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.279633    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:11.279821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.279748    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:16:11.379125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.379066    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scfcx\" (UniqueName: \"kubernetes.io/projected/4efb4d17-754a-4d67-addd-cefbe39d96ed-kube-api-access-scfcx\") pod \"maas-api-revoke-sub-d5461e71-mwj28\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.380738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.380707    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5461e71-mwj28\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.381103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.381079    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5461e71-mwj28\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.485878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.482407    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5461e71-mwj28\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.485878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.482651    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfcx\" (UniqueName: \"kubernetes.io/projected/4efb4d17-754a-4d67-addd-cefbe39d96ed-kube-api-access-scfcx\") pod \"maas-api-revoke-sub-d5461e71-mwj28\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.485878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.482759    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5461e71-mwj28\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.485878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.484492    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5461e71-mwj28\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.506395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.506350    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5461e71-mwj28\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.520670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.520591    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfcx\" (UniqueName: \"kubernetes.io/projected/4efb4d17-754a-4d67-addd-cefbe39d96ed-kube-api-access-scfcx\") pod \"maas-api-revoke-sub-d5461e71-mwj28\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.612209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.612164    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:11.941989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:11.941956    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"]
Oct 30 21:16:11.952311 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:11.952280    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efb4d17_754a_4d67_addd_cefbe39d96ed.slice/crio-208e36d2e3a840cc545150adea0e8e835b50b4e43343713b406a47bd2ce094d3 WatchSource:0}: Error finding container 208e36d2e3a840cc545150adea0e8e835b50b4e43343713b406a47bd2ce094d3: Status 404 returned error can't find the container with id 208e36d2e3a840cc545150adea0e8e835b50b4e43343713b406a47bd2ce094d3
Oct 30 21:16:12.108186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.107991    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"]
Oct 30 21:16:12.114875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.114833    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.120430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.120379    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:16:12.120545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.120489    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:16:12.120721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.120698    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:16:12.120808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.120776    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:16:12.120889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.120869    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b2215cc9b089\""
Oct 30 21:16:12.120981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.120967    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-sfdtt\""
Oct 30 21:16:12.124320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.124179    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"]
Oct 30 21:16:12.203949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.201395    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-routing-overlay-volume\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.203949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.201516    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-plugins-config-volume\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.203949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.201668    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25l66\" (UniqueName: \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-kube-api-access-25l66\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.203949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.201805    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-provider-credentials\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.350902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.315064    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l66\" (UniqueName: \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-kube-api-access-25l66\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.350902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.315245    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-provider-credentials\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.350902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.315334    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-routing-overlay-volume\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.350902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.315438    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-plugins-config-volume\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.350902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.316512    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-plugins-config-volume\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.359981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.359952    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25l66\" (UniqueName: \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-kube-api-access-25l66\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.361399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.361363    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-routing-overlay-volume\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.365082 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.365052    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-provider-credentials\") pod \"payload-processing-external-model-59cd9876d5-km8cr\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") " pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.433555 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.433520    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:12.679649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.679614    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28" event={"ID":"4efb4d17-754a-4d67-addd-cefbe39d96ed","Type":"ContainerStarted","Data":"2d33dded35176d5ebdaaaf42ed9ea4a78289fd5b98bc3f9ec9bf544398793437"}
Oct 30 21:16:12.679821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.679669    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28" event={"ID":"4efb4d17-754a-4d67-addd-cefbe39d96ed","Type":"ContainerStarted","Data":"208e36d2e3a840cc545150adea0e8e835b50b4e43343713b406a47bd2ce094d3"}
Oct 30 21:16:12.705827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.705767    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28" podStartSLOduration=1.705753388 podStartE2EDuration="1.705753388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:12.70042704 +0000 UTC m=+2339.192243594" watchObservedRunningTime="2026-10-30 21:16:12.705753388 +0000 UTC m=+2339.197569967"
Oct 30 21:16:12.749924 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:12.749853    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16213cf3_133d_44f0_b17e_83d102d07a36.slice/crio-2e8e8b56e33c1f20bfb244dd397c67af5fcbc6cc903317a42b417e105b16f080 WatchSource:0}: Error finding container 2e8e8b56e33c1f20bfb244dd397c67af5fcbc6cc903317a42b417e105b16f080: Status 404 returned error can't find the container with id 2e8e8b56e33c1f20bfb244dd397c67af5fcbc6cc903317a42b417e105b16f080
Oct 30 21:16:12.750318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:12.750298    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"]
Oct 30 21:16:13.695595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.695558    4810 generic.go:356] "Generic (PLEG): container finished" podID="4efb4d17-754a-4d67-addd-cefbe39d96ed" containerID="2d33dded35176d5ebdaaaf42ed9ea4a78289fd5b98bc3f9ec9bf544398793437" exitCode=0
Oct 30 21:16:13.696205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.695643    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28" event={"ID":"4efb4d17-754a-4d67-addd-cefbe39d96ed","Type":"ContainerDied","Data":"2d33dded35176d5ebdaaaf42ed9ea4a78289fd5b98bc3f9ec9bf544398793437"}
Oct 30 21:16:13.698770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.698743    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr" event={"ID":"16213cf3-133d-44f0-b17e-83d102d07a36","Type":"ContainerStarted","Data":"b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917"}
Oct 30 21:16:13.698913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.698792    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr" event={"ID":"16213cf3-133d-44f0-b17e-83d102d07a36","Type":"ContainerStarted","Data":"2e8e8b56e33c1f20bfb244dd397c67af5fcbc6cc903317a42b417e105b16f080"}
Oct 30 21:16:13.698913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.698836    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:13.711715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.711686    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"]
Oct 30 21:16:13.717432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.717414    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:13.741063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.741031    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"]
Oct 30 21:16:13.771575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.771517    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr" podStartSLOduration=1.771499549 podStartE2EDuration="1.771499549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:13.766215558 +0000 UTC m=+2340.258032102" watchObservedRunningTime="2026-10-30 21:16:13.771499549 +0000 UTC m=+2340.263316094"
Oct 30 21:16:13.842417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.842387    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1df15ae1-bbln6\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:13.842820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.842789    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1df15ae1-bbln6\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:13.842993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.842927    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjb6\" (UniqueName: \"kubernetes.io/projected/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-kube-api-access-vmjb6\") pod \"maas-api-revoke-sub-1df15ae1-bbln6\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:13.945009 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.944968    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1df15ae1-bbln6\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:13.945189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.945053    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjb6\" (UniqueName: \"kubernetes.io/projected/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-kube-api-access-vmjb6\") pod \"maas-api-revoke-sub-1df15ae1-bbln6\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:13.945189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.945136    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1df15ae1-bbln6\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:13.946445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.946383    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1df15ae1-bbln6\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:13.949251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.948332    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1df15ae1-bbln6\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:13.959532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:13.959509    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjb6\" (UniqueName: \"kubernetes.io/projected/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-kube-api-access-vmjb6\") pod \"maas-api-revoke-sub-1df15ae1-bbln6\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:14.035010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:14.034983    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:14.365010 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:14.364980    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6d5c43_5f90_4877_bdc5_5772a11c9d6d.slice/crio-d9741060a5fd7f3428057b3c52b0d0f275269dd3015b26ede4098255bd225792 WatchSource:0}: Error finding container d9741060a5fd7f3428057b3c52b0d0f275269dd3015b26ede4098255bd225792: Status 404 returned error can't find the container with id d9741060a5fd7f3428057b3c52b0d0f275269dd3015b26ede4098255bd225792
Oct 30 21:16:14.365402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:14.365379    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"]
Oct 30 21:16:14.712116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:14.712078    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6" event={"ID":"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d","Type":"ContainerStarted","Data":"d9741060a5fd7f3428057b3c52b0d0f275269dd3015b26ede4098255bd225792"}
Oct 30 21:16:14.923681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:14.923655    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:15.062250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.062211    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-openshift-ca\") pod \"4efb4d17-754a-4d67-addd-cefbe39d96ed\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") "
Oct 30 21:16:15.062421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.062257    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efb4d17-754a-4d67-addd-cefbe39d96ed-kube-api-access-scfcx\" (UniqueName: \"kubernetes.io/projected/4efb4d17-754a-4d67-addd-cefbe39d96ed-kube-api-access-scfcx\") pod \"4efb4d17-754a-4d67-addd-cefbe39d96ed\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") "
Oct 30 21:16:15.062421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.062350    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-xks-ca\") pod \"4efb4d17-754a-4d67-addd-cefbe39d96ed\" (UID: \"4efb4d17-754a-4d67-addd-cefbe39d96ed\") "
Oct 30 21:16:15.062642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.062614    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-openshift-ca" pod "4efb4d17-754a-4d67-addd-cefbe39d96ed" (UID: "4efb4d17-754a-4d67-addd-cefbe39d96ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:15.063030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.063009    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:15.064570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.064542    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-xks-ca" pod "4efb4d17-754a-4d67-addd-cefbe39d96ed" (UID: "4efb4d17-754a-4d67-addd-cefbe39d96ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:15.064570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.064557    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efb4d17-754a-4d67-addd-cefbe39d96ed-kube-api-access-scfcx" pod "4efb4d17-754a-4d67-addd-cefbe39d96ed" (UID: "4efb4d17-754a-4d67-addd-cefbe39d96ed"). InnerVolumeSpecName "kube-api-access-scfcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:15.164251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.164218    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scfcx\" (UniqueName: \"kubernetes.io/projected/4efb4d17-754a-4d67-addd-cefbe39d96ed-kube-api-access-scfcx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:15.164251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.164246    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4efb4d17-754a-4d67-addd-cefbe39d96ed-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:15.739585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.739494    4810 generic.go:356] "Generic (PLEG): container finished" podID="eb6d5c43-5f90-4877-bdc5-5772a11c9d6d" containerID="522517c01ffdb24e884d4314e7c7ee1cbfa3cb1adaf26b37b2869c51ef6e8d51" exitCode=0
Oct 30 21:16:15.740043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.739580    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6" event={"ID":"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d","Type":"ContainerDied","Data":"522517c01ffdb24e884d4314e7c7ee1cbfa3cb1adaf26b37b2869c51ef6e8d51"}
Oct 30 21:16:15.741803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.741777    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28" event={"ID":"4efb4d17-754a-4d67-addd-cefbe39d96ed","Type":"ContainerDied","Data":"208e36d2e3a840cc545150adea0e8e835b50b4e43343713b406a47bd2ce094d3"}
Oct 30 21:16:15.741916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.741807    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"
Oct 30 21:16:15.741916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:15.741826    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208e36d2e3a840cc545150adea0e8e835b50b4e43343713b406a47bd2ce094d3"
Oct 30 21:16:16.061223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:16.061142    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"]
Oct 30 21:16:16.853888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:16.853561    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"]
Oct 30 21:16:17.048241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.048214    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:17.227889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.224663    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-xks-ca\") pod \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") "
Oct 30 21:16:17.227889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.224740    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-openshift-ca\") pod \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") "
Oct 30 21:16:17.227889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.224799    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-kube-api-access-vmjb6\" (UniqueName: \"kubernetes.io/projected/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-kube-api-access-vmjb6\") pod \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\" (UID: \"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d\") "
Oct 30 21:16:17.242875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.238060    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-openshift-ca" pod "eb6d5c43-5f90-4877-bdc5-5772a11c9d6d" (UID: "eb6d5c43-5f90-4877-bdc5-5772a11c9d6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:17.265197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.265118    4810 scope.go:122] "RemoveContainer" containerID="336c48ab0ce2146ffa9fbd90e6c5583e1d8866d5b62d7abd853bf3f129400f6d"
Oct 30 21:16:17.277707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.277217    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-kube-api-access-vmjb6" pod "eb6d5c43-5f90-4877-bdc5-5772a11c9d6d" (UID: "eb6d5c43-5f90-4877-bdc5-5772a11c9d6d"). InnerVolumeSpecName "kube-api-access-vmjb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:17.317919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.312121    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-xks-ca" pod "eb6d5c43-5f90-4877-bdc5-5772a11c9d6d" (UID: "eb6d5c43-5f90-4877-bdc5-5772a11c9d6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:17.400059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.399101    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:17.400059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.399134    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:17.400059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.399151    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjb6\" (UniqueName: \"kubernetes.io/projected/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d-kube-api-access-vmjb6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:17.766023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.765993    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"
Oct 30 21:16:17.766196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.766024    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6" event={"ID":"eb6d5c43-5f90-4877-bdc5-5772a11c9d6d","Type":"ContainerDied","Data":"d9741060a5fd7f3428057b3c52b0d0f275269dd3015b26ede4098255bd225792"}
Oct 30 21:16:17.766196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:17.766068    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9741060a5fd7f3428057b3c52b0d0f275269dd3015b26ede4098255bd225792"
Oct 30 21:16:18.391196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:18.391162    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:16:21.061522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.061464    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp" podUID="1bac0e38-3d1a-438d-ae21-1186242471b9" containerName="payload-processing" containerID="cri-o://ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c" gracePeriod=25
Oct 30 21:16:21.345898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.345872    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:16:21.473435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.473398    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-kube-api-access-xdf6f\" (UniqueName: \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-kube-api-access-xdf6f\") pod \"1bac0e38-3d1a-438d-ae21-1186242471b9\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") "
Oct 30 21:16:21.473610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.473521    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-plugins-config-volume\") pod \"1bac0e38-3d1a-438d-ae21-1186242471b9\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") "
Oct 30 21:16:21.473694 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.473620    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-provider-credentials\") pod \"1bac0e38-3d1a-438d-ae21-1186242471b9\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") "
Oct 30 21:16:21.473810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.473789    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-routing-overlay-volume\") pod \"1bac0e38-3d1a-438d-ae21-1186242471b9\" (UID: \"1bac0e38-3d1a-438d-ae21-1186242471b9\") "
Oct 30 21:16:21.474816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.474768    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-routing-overlay-volume" pod "1bac0e38-3d1a-438d-ae21-1186242471b9" (UID: "1bac0e38-3d1a-438d-ae21-1186242471b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:21.474957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.473994    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-plugins-config-volume" pod "1bac0e38-3d1a-438d-ae21-1186242471b9" (UID: "1bac0e38-3d1a-438d-ae21-1186242471b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:21.475755 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.475418    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:21.475755 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.475445    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bac0e38-3d1a-438d-ae21-1186242471b9-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:21.490963 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.490927    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-provider-credentials" pod "1bac0e38-3d1a-438d-ae21-1186242471b9" (UID: "1bac0e38-3d1a-438d-ae21-1186242471b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:21.491106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.490956    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-kube-api-access-xdf6f" pod "1bac0e38-3d1a-438d-ae21-1186242471b9" (UID: "1bac0e38-3d1a-438d-ae21-1186242471b9"). InnerVolumeSpecName "kube-api-access-xdf6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:21.576603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.576532    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdf6f\" (UniqueName: \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-kube-api-access-xdf6f\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:21.576603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.576557    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bac0e38-3d1a-438d-ae21-1186242471b9-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:21.800917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.800880    4810 generic.go:356] "Generic (PLEG): container finished" podID="1bac0e38-3d1a-438d-ae21-1186242471b9" containerID="ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c" exitCode=0
Oct 30 21:16:21.801083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.800928    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp" event={"ID":"1bac0e38-3d1a-438d-ae21-1186242471b9","Type":"ContainerDied","Data":"ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c"}
Oct 30 21:16:21.801083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.800946    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"
Oct 30 21:16:21.801083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.800980    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp" event={"ID":"1bac0e38-3d1a-438d-ae21-1186242471b9","Type":"ContainerDied","Data":"e9518da2ae9fc57b779551361b5628a1e38f6d61fc39a18d0f5074d7955729cc"}
Oct 30 21:16:21.801083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.801019    4810 scope.go:122] "RemoveContainer" containerID="ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c"
Oct 30 21:16:21.812587 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.812568    4810 scope.go:122] "RemoveContainer" containerID="ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c"
Oct 30 21:16:21.812874 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:21.812838    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c\": container with ID starting with ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c not found: ID does not exist" containerID="ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c"
Oct 30 21:16:21.812938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.812884    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c"} err="failed to get container status \"ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c\": rpc error: code = NotFound desc = could not find container \"ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c\": container with ID starting with ddcb7c2dcb59c14c3a4b3269187b2924f8dfa7d1eb83406e267c0146dece5c0c not found: ID does not exist"
Oct 30 21:16:21.825930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.825900    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"]
Oct 30 21:16:21.833131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.833070    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-00909f/payload-processing-external-model-e2e-extmod-a-00909f-85b4zjbbp"]
Oct 30 21:16:21.864301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:21.856618    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv" podUID="1dfbdf2a-9bad-4968-acb2-4759ca868815" containerName="payload-processing" containerID="cri-o://9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0" gracePeriod=25
Oct 30 21:16:22.196567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.196528    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:16:22.310887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.291374    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-routing-overlay-volume\") pod \"1dfbdf2a-9bad-4968-acb2-4759ca868815\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") "
Oct 30 21:16:22.311232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.296961    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-routing-overlay-volume" pod "1dfbdf2a-9bad-4968-acb2-4759ca868815" (UID: "1dfbdf2a-9bad-4968-acb2-4759ca868815"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:22.311362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.311270    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-kube-api-access-xs922\" (UniqueName: \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-kube-api-access-xs922\") pod \"1dfbdf2a-9bad-4968-acb2-4759ca868815\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") "
Oct 30 21:16:22.311362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.311313    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-provider-credentials\") pod \"1dfbdf2a-9bad-4968-acb2-4759ca868815\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") "
Oct 30 21:16:22.311844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.311487    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-plugins-config-volume\") pod \"1dfbdf2a-9bad-4968-acb2-4759ca868815\" (UID: \"1dfbdf2a-9bad-4968-acb2-4759ca868815\") "
Oct 30 21:16:22.366928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.312209    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:22.392443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.312611    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-plugins-config-volume" pod "1dfbdf2a-9bad-4968-acb2-4759ca868815" (UID: "1dfbdf2a-9bad-4968-acb2-4759ca868815"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:22.392647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.321150    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-kube-api-access-xs922" pod "1dfbdf2a-9bad-4968-acb2-4759ca868815" (UID: "1dfbdf2a-9bad-4968-acb2-4759ca868815"). InnerVolumeSpecName "kube-api-access-xs922". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:22.404114 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.353027    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-provider-credentials" pod "1dfbdf2a-9bad-4968-acb2-4759ca868815" (UID: "1dfbdf2a-9bad-4968-acb2-4759ca868815"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.416944    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"]
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418227    4810 cpu_manager.go:414] "containerMap: removing container" podUID="eb6d5c43-5f90-4877-bdc5-5772a11c9d6d" containerName="revoke-keys"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418248    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb6d5c43-5f90-4877-bdc5-5772a11c9d6d" containerName="revoke-keys"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418324    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1bac0e38-3d1a-438d-ae21-1186242471b9" containerName="payload-processing"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418333    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bac0e38-3d1a-438d-ae21-1186242471b9" containerName="payload-processing"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418345    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4efb4d17-754a-4d67-addd-cefbe39d96ed" containerName="revoke-keys"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418354    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efb4d17-754a-4d67-addd-cefbe39d96ed" containerName="revoke-keys"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418376    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1dfbdf2a-9bad-4968-acb2-4759ca868815" containerName="payload-processing"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418383    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfbdf2a-9bad-4968-acb2-4759ca868815" containerName="payload-processing"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418546    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb6d5c43-5f90-4877-bdc5-5772a11c9d6d" containerName="revoke-keys"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418568    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bac0e38-3d1a-438d-ae21-1186242471b9" containerName="payload-processing"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418578    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efb4d17-754a-4d67-addd-cefbe39d96ed" containerName="revoke-keys"
Oct 30 21:16:22.421125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.418592    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfbdf2a-9bad-4968-acb2-4759ca868815" containerName="payload-processing"
Oct 30 21:16:22.463882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.454323    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"]
Oct 30 21:16:22.463882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.454429    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.501256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.501215    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.501441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.501287    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stbgk\" (UniqueName: \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-kube-api-access-stbgk\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.501529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.501453    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.501587 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.501573    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.501703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.501684    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs922\" (UniqueName: \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-kube-api-access-xs922\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:22.501766 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.501709    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1dfbdf2a-9bad-4968-acb2-4759ca868815-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:22.501766 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.501724    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfbdf2a-9bad-4968-acb2-4759ca868815-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:22.602816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.602780    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.603006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.602829    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stbgk\" (UniqueName: \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-kube-api-access-stbgk\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.603006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.602923    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.603006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.602984    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.603588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.603561    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.603751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.603725    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.605406 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.605385    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.611537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.611510    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stbgk\" (UniqueName: \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-kube-api-access-stbgk\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.780901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.774437    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"]
Oct 30 21:16:22.784878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.784702    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:22.788952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.788881    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:22.789629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.789543    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:16:22.809903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.807701    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e3a0111-4l2mp\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:22.828879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.813885    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e3a0111-4l2mp\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:22.828879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.813977    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74ll\" (UniqueName: \"kubernetes.io/projected/61d2e826-c5fc-4832-8fff-31f6cd1e0561-kube-api-access-h74ll\") pod \"maas-api-revoke-sub-4e3a0111-4l2mp\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:22.828879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.826506    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:22.859892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.853351    4810 generic.go:356] "Generic (PLEG): container finished" podID="1dfbdf2a-9bad-4968-acb2-4759ca868815" containerID="9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0" exitCode=0
Oct 30 21:16:22.859892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.853547    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv" event={"ID":"1dfbdf2a-9bad-4968-acb2-4759ca868815","Type":"ContainerDied","Data":"9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0"}
Oct 30 21:16:22.859892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.853599    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv" event={"ID":"1dfbdf2a-9bad-4968-acb2-4759ca868815","Type":"ContainerDied","Data":"6a3926f4f1a5e7b79f1ffda6629b9d94c98c449e2d130f8673b6845c003123de"}
Oct 30 21:16:22.859892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.853629    4810 scope.go:122] "RemoveContainer" containerID="9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0"
Oct 30 21:16:22.859892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.853746    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"
Oct 30 21:16:22.896878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.894657    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"]
Oct 30 21:16:22.907107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.907068    4810 scope.go:122] "RemoveContainer" containerID="9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0"
Oct 30 21:16:22.907596 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:22.907569    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0\": container with ID starting with 9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0 not found: ID does not exist" containerID="9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0"
Oct 30 21:16:22.907713 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.907609    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0"} err="failed to get container status \"9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0\": rpc error: code = NotFound desc = could not find container \"9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0\": container with ID starting with 9b05851f8e9121fb4cb2000d0ddaaafed82b216be8e80f0c87caf9f03a6bb7a0 not found: ID does not exist"
Oct 30 21:16:22.925895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.923308    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e3a0111-4l2mp\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:22.925895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.923398    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74ll\" (UniqueName: \"kubernetes.io/projected/61d2e826-c5fc-4832-8fff-31f6cd1e0561-kube-api-access-h74ll\") pod \"maas-api-revoke-sub-4e3a0111-4l2mp\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:22.925895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.923465    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e3a0111-4l2mp\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:22.925895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.925364    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e3a0111-4l2mp\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:22.939680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.939608    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e3a0111-4l2mp\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:22.942463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.942410    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74ll\" (UniqueName: \"kubernetes.io/projected/61d2e826-c5fc-4832-8fff-31f6cd1e0561-kube-api-access-h74ll\") pod \"maas-api-revoke-sub-4e3a0111-4l2mp\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:22.948092 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.947696    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"]
Oct 30 21:16:22.953685 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:22.953103    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-345bc4/payload-processing-external-model-e2e-extmod-a-345bc4-7b96cskrv"]
Oct 30 21:16:23.146977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:23.146945    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:23.236676 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:23.236632    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d9a5cf_731a_4ad4_a00c_39cf4b26ca3c.slice/crio-135e5a35d66de9be322f0c7bc2734906f100ec72d59884b82e0a4aca7c04c49a WatchSource:0}: Error finding container 135e5a35d66de9be322f0c7bc2734906f100ec72d59884b82e0a4aca7c04c49a: Status 404 returned error can't find the container with id 135e5a35d66de9be322f0c7bc2734906f100ec72d59884b82e0a4aca7c04c49a
Oct 30 21:16:23.244459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:23.244432    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"]
Oct 30 21:16:23.438298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:23.438270    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"]
Oct 30 21:16:23.439174 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:23.439145    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d2e826_c5fc_4832_8fff_31f6cd1e0561.slice/crio-5a88172a15fc6d889dfd96b13acc4bcd7e8cfaf853bf5a184d202b850c9dccc0 WatchSource:0}: Error finding container 5a88172a15fc6d889dfd96b13acc4bcd7e8cfaf853bf5a184d202b850c9dccc0: Status 404 returned error can't find the container with id 5a88172a15fc6d889dfd96b13acc4bcd7e8cfaf853bf5a184d202b850c9dccc0
Oct 30 21:16:23.666574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:23.666536    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bac0e38-3d1a-438d-ae21-1186242471b9" path="/var/lib/kubelet/pods/1bac0e38-3d1a-438d-ae21-1186242471b9/volumes"
Oct 30 21:16:23.667208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:23.667189    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfbdf2a-9bad-4968-acb2-4759ca868815" path="/var/lib/kubelet/pods/1dfbdf2a-9bad-4968-acb2-4759ca868815/volumes"
Oct 30 21:16:23.901146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:23.899975    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp" event={"ID":"61d2e826-c5fc-4832-8fff-31f6cd1e0561","Type":"ContainerStarted","Data":"5a88172a15fc6d889dfd96b13acc4bcd7e8cfaf853bf5a184d202b850c9dccc0"}
Oct 30 21:16:23.905683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:23.905649    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c" event={"ID":"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c","Type":"ContainerStarted","Data":"1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01"}
Oct 30 21:16:23.907917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:23.905701    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c" event={"ID":"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c","Type":"ContainerStarted","Data":"135e5a35d66de9be322f0c7bc2734906f100ec72d59884b82e0a4aca7c04c49a"}
Oct 30 21:16:23.907917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:23.905759    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:24.716432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:24.716340    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:24.748592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:24.748546    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c" podStartSLOduration=2.748531335 podStartE2EDuration="2.748531335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:23.929020638 +0000 UTC m=+2350.420837182" watchObservedRunningTime="2026-10-30 21:16:24.748531335 +0000 UTC m=+2351.240347930"
Oct 30 21:16:24.929881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:24.929833    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"]
Oct 30 21:16:24.968931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:24.968644    4810 generic.go:356] "Generic (PLEG): container finished" podID="61d2e826-c5fc-4832-8fff-31f6cd1e0561" containerID="dfde8135d1468102329852696cec9695a15bf6aefb224ad95c08bbb4912abd0f" exitCode=0
Oct 30 21:16:25.129320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.129276    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp" event={"ID":"61d2e826-c5fc-4832-8fff-31f6cd1e0561","Type":"ContainerDied","Data":"dfde8135d1468102329852696cec9695a15bf6aefb224ad95c08bbb4912abd0f"}
Oct 30 21:16:25.129571 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.129345    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"]
Oct 30 21:16:25.129571 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.129450    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.183224 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:25.183188    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:16:25.226166 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.226082    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-474v8\" (UniqueName: \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-kube-api-access-474v8\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.226166 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.226132    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.226401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.226299    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.226401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.226367    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.328610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.328558    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.328831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.328652    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.329010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.328988    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-474v8\" (UniqueName: \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-kube-api-access-474v8\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.329090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.329062    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.330133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.330107    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.330703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.330682    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.334562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.334537    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.344635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.344608    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-474v8\" (UniqueName: \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-kube-api-access-474v8\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.456410 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.456371    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:25.777908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.773608    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"]
Oct 30 21:16:25.991446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.991390    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb" event={"ID":"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82","Type":"ContainerStarted","Data":"44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846"}
Oct 30 21:16:25.991635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.991488    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb" event={"ID":"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82","Type":"ContainerStarted","Data":"3b2fdde9847865bef99643fdcf3f041e6c46a5c885c3ea33699cac53efea7bea"}
Oct 30 21:16:25.991847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:25.991705    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:26.016000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.015941    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb" podStartSLOduration=2.015921666 podStartE2EDuration="2.015921666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:26.009051457 +0000 UTC m=+2352.500868003" watchObservedRunningTime="2026-10-30 21:16:26.015921666 +0000 UTC m=+2352.507738212"
Oct 30 21:16:26.143045 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:26.143008    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-a082f645dd4a: secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:26.143045 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:26.143047    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb: secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:26.143269 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:26.143119    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials podName:d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:26.643099338 +0000 UTC m=+2353.134915878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb" (UID: "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82") : secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:26.155617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.155592    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:26.243934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.243851    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-openshift-ca\") pod \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") "
Oct 30 21:16:26.244146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.243954    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d2e826-c5fc-4832-8fff-31f6cd1e0561-kube-api-access-h74ll\" (UniqueName: \"kubernetes.io/projected/61d2e826-c5fc-4832-8fff-31f6cd1e0561-kube-api-access-h74ll\") pod \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") "
Oct 30 21:16:26.244146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.244056    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-xks-ca\") pod \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\" (UID: \"61d2e826-c5fc-4832-8fff-31f6cd1e0561\") "
Oct 30 21:16:26.244320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.244291    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-openshift-ca" pod "61d2e826-c5fc-4832-8fff-31f6cd1e0561" (UID: "61d2e826-c5fc-4832-8fff-31f6cd1e0561"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:26.244708 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.244689    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:26.246134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.246107    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d2e826-c5fc-4832-8fff-31f6cd1e0561-kube-api-access-h74ll" pod "61d2e826-c5fc-4832-8fff-31f6cd1e0561" (UID: "61d2e826-c5fc-4832-8fff-31f6cd1e0561"). InnerVolumeSpecName "kube-api-access-h74ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:26.246285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.246146    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-xks-ca" pod "61d2e826-c5fc-4832-8fff-31f6cd1e0561" (UID: "61d2e826-c5fc-4832-8fff-31f6cd1e0561"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:26.261019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.260992    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"]
Oct 30 21:16:26.298707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.298464    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"]
Oct 30 21:16:26.300307 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.300276    4810 cpu_manager.go:414] "containerMap: removing container" podUID="61d2e826-c5fc-4832-8fff-31f6cd1e0561" containerName="revoke-keys"
Oct 30 21:16:26.300307 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.300306    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d2e826-c5fc-4832-8fff-31f6cd1e0561" containerName="revoke-keys"
Oct 30 21:16:26.302121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.302093    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d2e826-c5fc-4832-8fff-31f6cd1e0561" containerName="revoke-keys"
Oct 30 21:16:26.308218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.308194    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.311637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.311578    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"]
Oct 30 21:16:26.346117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.346082    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d2e826-c5fc-4832-8fff-31f6cd1e0561-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:26.346117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.346118    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h74ll\" (UniqueName: \"kubernetes.io/projected/61d2e826-c5fc-4832-8fff-31f6cd1e0561-kube-api-access-h74ll\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:26.371198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.371170    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"]
Oct 30 21:16:26.407884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.405084    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.420879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.420828    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"]
Oct 30 21:16:26.450120 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.450052    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.450322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.450229    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.450322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.450280    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.450436 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.450320    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c91a97f8-68pnd\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.450489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.450471    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c91a97f8-68pnd\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.450567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.450548    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xc9\" (UniqueName: \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-kube-api-access-99xc9\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.450810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.450782    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rqq\" (UniqueName: \"kubernetes.io/projected/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-kube-api-access-h8rqq\") pod \"maas-api-revoke-sub-c91a97f8-68pnd\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.552211 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.552174    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.552377 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.552235    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.552377 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.552266    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.552377 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.552294    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c91a97f8-68pnd\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.552567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.552389    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c91a97f8-68pnd\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.552567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.552459    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xc9\" (UniqueName: \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-kube-api-access-99xc9\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.552567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.552526    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rqq\" (UniqueName: \"kubernetes.io/projected/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-kube-api-access-h8rqq\") pod \"maas-api-revoke-sub-c91a97f8-68pnd\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.553015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.552984    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.553157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.553136    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.553224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.553138    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c91a97f8-68pnd\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.555237 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.555213    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c91a97f8-68pnd\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.555338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.555294    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.561673 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.561619    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xc9\" (UniqueName: \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-kube-api-access-99xc9\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.562006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.561986    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rqq\" (UniqueName: \"kubernetes.io/projected/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-kube-api-access-h8rqq\") pod \"maas-api-revoke-sub-c91a97f8-68pnd\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.630070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.630035    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:26.654629 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:26.654602    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-a082f645dd4a: secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:26.654629 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:26.654631    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb: secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:26.654772 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:26.654688    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials podName:d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:27.654674198 +0000 UTC m=+2354.146490721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb" (UID: "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82") : secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:26.763092 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.763054    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:26.917431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:26.917398    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"]
Oct 30 21:16:26.917431 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:26.917418    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e6f191_eaf5_4d65_834a_10176982b9f7.slice/crio-67c1517885f5bf7970879484b7ba795d0a5e292e76a3534e22943851ee7b2949 WatchSource:0}: Error finding container 67c1517885f5bf7970879484b7ba795d0a5e292e76a3534e22943851ee7b2949: Status 404 returned error can't find the container with id 67c1517885f5bf7970879484b7ba795d0a5e292e76a3534e22943851ee7b2949
Oct 30 21:16:27.004967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:27.004944    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"
Oct 30 21:16:27.005118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:27.004981    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp" event={"ID":"61d2e826-c5fc-4832-8fff-31f6cd1e0561","Type":"ContainerDied","Data":"5a88172a15fc6d889dfd96b13acc4bcd7e8cfaf853bf5a184d202b850c9dccc0"}
Oct 30 21:16:27.005118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:27.005038    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a88172a15fc6d889dfd96b13acc4bcd7e8cfaf853bf5a184d202b850c9dccc0"
Oct 30 21:16:27.007101 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:27.007054    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm" event={"ID":"80e6f191-eaf5-4d65-834a-10176982b9f7","Type":"ContainerStarted","Data":"186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78"}
Oct 30 21:16:27.007377 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:27.007125    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm" event={"ID":"80e6f191-eaf5-4d65-834a-10176982b9f7","Type":"ContainerStarted","Data":"67c1517885f5bf7970879484b7ba795d0a5e292e76a3534e22943851ee7b2949"}
Oct 30 21:16:27.007377 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:27.007364    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:27.024903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:27.024844    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm" podStartSLOduration=1.024830125 podStartE2EDuration="1.024830125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:27.023733066 +0000 UTC m=+2353.515549608" watchObservedRunningTime="2026-10-30 21:16:27.024830125 +0000 UTC m=+2353.516646718"
Oct 30 21:16:27.048880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:27.048837    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"]
Oct 30 21:16:27.049068 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:27.049048    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode860e0db_5f7e_4cb0_8a17_d1fa7d3edf17.slice/crio-a87b329182f0e509b498a2d7b32c4c652c3a105999c5eb0052d851e4e04226c4 WatchSource:0}: Error finding container a87b329182f0e509b498a2d7b32c4c652c3a105999c5eb0052d851e4e04226c4: Status 404 returned error can't find the container with id a87b329182f0e509b498a2d7b32c4c652c3a105999c5eb0052d851e4e04226c4
Oct 30 21:16:27.690045 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:27.690008    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-a082f645dd4a: secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:27.690045 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:27.690041    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb: secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:27.690288 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:27.690098    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials podName:d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:29.69008166 +0000 UTC m=+2356.181898185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb" (UID: "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82") : secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:28.018780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:28.018728    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd" event={"ID":"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17","Type":"ContainerStarted","Data":"94be1b8e87ab6b720f0e1c8106dee2666e323fb292d4c75b9a52da2ee9376070"}
Oct 30 21:16:28.019779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:28.019745    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd" event={"ID":"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17","Type":"ContainerStarted","Data":"a87b329182f0e509b498a2d7b32c4c652c3a105999c5eb0052d851e4e04226c4"}
Oct 30 21:16:28.043776 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:28.043726    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd" podStartSLOduration=2.043705991 podStartE2EDuration="2.043705991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:28.037073416 +0000 UTC m=+2354.528889961" watchObservedRunningTime="2026-10-30 21:16:28.043705991 +0000 UTC m=+2354.535522535"
Oct 30 21:16:29.031413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:29.031373    4810 generic.go:356] "Generic (PLEG): container finished" podID="e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17" containerID="94be1b8e87ab6b720f0e1c8106dee2666e323fb292d4c75b9a52da2ee9376070" exitCode=0
Oct 30 21:16:29.031806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:29.031425    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd" event={"ID":"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17","Type":"ContainerDied","Data":"94be1b8e87ab6b720f0e1c8106dee2666e323fb292d4c75b9a52da2ee9376070"}
Oct 30 21:16:29.720934 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:29.720887    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-a082f645dd4a: secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:29.720934 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:29.720925    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb: secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:29.721181 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:29.720986    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials podName:d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:33.72096829 +0000 UTC m=+2360.212784813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb" (UID: "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82") : secret "agc-cred-a082f645dd4a" not found
Oct 30 21:16:30.220043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:30.220013    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:30.330911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:30.330768    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-kube-api-access-h8rqq\" (UniqueName: \"kubernetes.io/projected/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-kube-api-access-h8rqq\") pod \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") "
Oct 30 21:16:30.332515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:30.332144    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-xks-ca\") pod \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") "
Oct 30 21:16:30.332515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:30.332236    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-openshift-ca\") pod \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\" (UID: \"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17\") "
Oct 30 21:16:30.332807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:30.332624    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-openshift-ca" pod "e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17" (UID: "e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:30.332988 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:30.332968    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:30.341760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:30.341706    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-xks-ca" pod "e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17" (UID: "e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:30.343012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:30.342981    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-kube-api-access-h8rqq" pod "e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17" (UID: "e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17"). InnerVolumeSpecName "kube-api-access-h8rqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:30.433907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:30.433873    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:30.433907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:30.433907    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rqq\" (UniqueName: \"kubernetes.io/projected/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17-kube-api-access-h8rqq\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:31.050414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:31.050383    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"
Oct 30 21:16:31.050414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:31.050405    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd" event={"ID":"e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17","Type":"ContainerDied","Data":"a87b329182f0e509b498a2d7b32c4c652c3a105999c5eb0052d851e4e04226c4"}
Oct 30 21:16:31.050632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:31.050452    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a87b329182f0e509b498a2d7b32c4c652c3a105999c5eb0052d851e4e04226c4"
Oct 30 21:16:32.238062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.238026    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"]
Oct 30 21:16:32.238902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.238885    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17" containerName="revoke-keys"
Oct 30 21:16:32.238902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.238902    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17" containerName="revoke-keys"
Oct 30 21:16:32.239088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.239078    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17" containerName="revoke-keys"
Oct 30 21:16:32.243565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.243544    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.247812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.247775    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7407c8c9a9d5\""
Oct 30 21:16:32.277053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.276936    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"]
Oct 30 21:16:32.278793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.278761    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-plugins-config-volume\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.279430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.279403    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-provider-credentials\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.279607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.279588    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-routing-overlay-volume\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.279815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.279795    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn24v\" (UniqueName: \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-kube-api-access-xn24v\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.381711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.381675    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn24v\" (UniqueName: \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-kube-api-access-xn24v\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.381914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.381897    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-plugins-config-volume\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.382023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.382007    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-provider-credentials\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.382081 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.382068    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-routing-overlay-volume\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.382782 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.382757    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-routing-overlay-volume\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.382910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.382766    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-plugins-config-volume\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.384737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.384713    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-provider-credentials\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.390315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.390292    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn24v\" (UniqueName: \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-kube-api-access-xn24v\") pod \"payload-processing-external-model-6494c5b757-s2z9w\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") " pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:32.564180 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:32.564101    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:33.019356 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.019298    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb" podUID="d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82" containerName="payload-processing" containerID="cri-o://44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846" gracePeriod=25
Oct 30 21:16:33.067626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.067597    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"]
Oct 30 21:16:33.113089 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:33.113057    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a2e74d_3e80_4667_a322_fa39e0b985ea.slice/crio-38cb2db17af0f4b6ec7edb3e36474207cf4dd570e2c6b6b8ecc5b5e8fb1dfcfc WatchSource:0}: Error finding container 38cb2db17af0f4b6ec7edb3e36474207cf4dd570e2c6b6b8ecc5b5e8fb1dfcfc: Status 404 returned error can't find the container with id 38cb2db17af0f4b6ec7edb3e36474207cf4dd570e2c6b6b8ecc5b5e8fb1dfcfc
Oct 30 21:16:33.304608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.304576    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:33.500936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.500897    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-routing-overlay-volume\") pod \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") "
Oct 30 21:16:33.501143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.501002    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-kube-api-access-474v8\" (UniqueName: \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-kube-api-access-474v8\") pod \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") "
Oct 30 21:16:33.501143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.501059    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials\") pod \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") "
Oct 30 21:16:33.501143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.501098    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-plugins-config-volume\") pod \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\" (UID: \"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82\") "
Oct 30 21:16:33.501639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.501608    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-routing-overlay-volume" pod "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82" (UID: "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:33.501840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.501804    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-plugins-config-volume" pod "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82" (UID: "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:33.502016 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.501948    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:33.503371 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.503312    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-kube-api-access-474v8" pod "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82" (UID: "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82"). InnerVolumeSpecName "kube-api-access-474v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:33.503371 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.503356    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials" pod "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82" (UID: "d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:33.604552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.603183    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-474v8\" (UniqueName: \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-kube-api-access-474v8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:33.604552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.603223    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:33.604552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:33.603238    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:34.094914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.091795    4810 generic.go:356] "Generic (PLEG): container finished" podID="d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82" containerID="44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846" exitCode=0
Oct 30 21:16:34.094914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.091930    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb" event={"ID":"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82","Type":"ContainerDied","Data":"44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846"}
Oct 30 21:16:34.094914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.091987    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb" event={"ID":"d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82","Type":"ContainerDied","Data":"3b2fdde9847865bef99643fdcf3f041e6c46a5c885c3ea33699cac53efea7bea"}
Oct 30 21:16:34.094914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.092018    4810 scope.go:122] "RemoveContainer" containerID="44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846"
Oct 30 21:16:34.094914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.092170    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"
Oct 30 21:16:34.100837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.100806    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w" event={"ID":"50a2e74d-3e80-4667-a322-fa39e0b985ea","Type":"ContainerStarted","Data":"7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961"}
Oct 30 21:16:34.101047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.100895    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w" event={"ID":"50a2e74d-3e80-4667-a322-fa39e0b985ea","Type":"ContainerStarted","Data":"38cb2db17af0f4b6ec7edb3e36474207cf4dd570e2c6b6b8ecc5b5e8fb1dfcfc"}
Oct 30 21:16:34.101226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.101125    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:34.113077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.113051    4810 scope.go:122] "RemoveContainer" containerID="44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846"
Oct 30 21:16:34.113488 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:34.113423    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846\": container with ID starting with 44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846 not found: ID does not exist" containerID="44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846"
Oct 30 21:16:34.113488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.113462    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846"} err="failed to get container status \"44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846\": rpc error: code = NotFound desc = could not find container \"44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846\": container with ID starting with 44d471213e8d651b540646c83d7f7434ec26c85fdf090c6e7c8882da38d1f846 not found: ID does not exist"
Oct 30 21:16:34.125567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.125539    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"]
Oct 30 21:16:34.132874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.131598    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-59cfxr6bb"]
Oct 30 21:16:34.148369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.148321    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w" podStartSLOduration=2.148304941 podStartE2EDuration="2.148304941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:34.142142724 +0000 UTC m=+2360.633959272" watchObservedRunningTime="2026-10-30 21:16:34.148304941 +0000 UTC m=+2360.640121486"
Oct 30 21:16:34.490086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.489964    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"]
Oct 30 21:16:34.528633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.528600    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"]
Oct 30 21:16:34.529441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.529425    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82" containerName="payload-processing"
Oct 30 21:16:34.529441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.529441    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82" containerName="payload-processing"
Oct 30 21:16:34.529593 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.529583    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82" containerName="payload-processing"
Oct 30 21:16:34.533549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.533534    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.539580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.539555    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b4eff0c43a87\""
Oct 30 21:16:34.544889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.544437    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"]
Oct 30 21:16:34.622754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.622716    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-provider-credentials\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.622980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.622791    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-routing-overlay-volume\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.623035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.622981    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhdl\" (UniqueName: \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-kube-api-access-4mhdl\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.623077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.623037    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-plugins-config-volume\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.724573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.724533    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhdl\" (UniqueName: \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-kube-api-access-4mhdl\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.724778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.724729    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-plugins-config-volume\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.725950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.725391    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-provider-credentials\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.725950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.725501    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-routing-overlay-volume\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.725950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.725920    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-plugins-config-volume\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.726332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.726311    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-routing-overlay-volume\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.728405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.728386    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-provider-credentials\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.737066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.737042    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhdl\" (UniqueName: \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-kube-api-access-4mhdl\") pod \"payload-processing-external-model-6b4fdd8ddc-ctw4d\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") " pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:34.854534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:34.854436    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:35.027613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:35.026323    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:35.205878 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:35.191380    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e1e0f5/agc-cred-be61d10a5f0a: secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:35.205878 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:35.191410    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c: secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:35.205878 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:35.191510    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials podName:67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:35.691474366 +0000 UTC m=+2362.183290908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c" (UID: "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c") : secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:35.213624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:35.213532    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"]
Oct 30 21:16:35.364266 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:35.364138    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"]
Oct 30 21:16:35.387091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:35.384462    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"]
Oct 30 21:16:35.696828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:35.695918    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82" path="/var/lib/kubelet/pods/d5d4dac2-ad0d-41fb-9e32-8bdf72e56c82/volumes"
Oct 30 21:16:35.710284 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:35.710244    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e1e0f5/agc-cred-be61d10a5f0a: secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:35.710284 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:35.710272    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c: secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:35.710559 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:35.710327    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials podName:67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:36.710311489 +0000 UTC m=+2363.202128030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c" (UID: "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c") : secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:36.244366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:36.244330    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d" event={"ID":"64069fea-0ef4-4e09-987c-0ebc2c9d8298","Type":"ContainerStarted","Data":"83b07498c680d1823e479dbd377fce61956e4d96c2de8b2c9711867af6d5f392"}
Oct 30 21:16:36.244565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:36.244383    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d" event={"ID":"64069fea-0ef4-4e09-987c-0ebc2c9d8298","Type":"ContainerStarted","Data":"759c6fae3352938142e00b7a2f0f848623366960150f64b80c0d54ae075a23a1"}
Oct 30 21:16:36.244565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:36.244437    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:36.265904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:36.265840    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d" podStartSLOduration=2.2658266 podStartE2EDuration="2.2658266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:36.261278274 +0000 UTC m=+2362.753094820" watchObservedRunningTime="2026-10-30 21:16:36.2658266 +0000 UTC m=+2362.757643144"
Oct 30 21:16:36.726392 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:36.726357    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e1e0f5/agc-cred-be61d10a5f0a: secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:36.726392 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:36.726382    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c: secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:36.726940 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:36.726435    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials podName:67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:38.726421406 +0000 UTC m=+2365.218237930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c" (UID: "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c") : secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:37.530704 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:37.530662    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"]
Oct 30 21:16:37.550168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:37.550135    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:37.561091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:37.561055    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"]
Oct 30 21:16:38.609785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.609744    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"]
Oct 30 21:16:38.616008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.615970    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.695928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.655478    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"]
Oct 30 21:16:38.700227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.700155    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"agc-cred-fcb5e5b64832\""
Oct 30 21:16:38.700430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.700319    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"payload-processing-external-model-e2e--789fdc40-dockercfg-jltsf\""
Oct 30 21:16:38.818964 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.818925    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kx6z\" (UniqueName: \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-kube-api-access-6kx6z\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.819214 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:38.819146    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e1e0f5/agc-cred-be61d10a5f0a: secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:38.819326 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:38.819274    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c: secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:38.819416 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:38.819338    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials podName:67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:42.819316782 +0000 UTC m=+2369.311133316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c" (UID: "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c") : secret "agc-cred-be61d10a5f0a" not found
Oct 30 21:16:38.819416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.819220    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.819589 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.819537    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.819672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.819590    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.924203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.924152    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.926150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.926115    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.927158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.927135    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kx6z\" (UniqueName: \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-kube-api-access-6kx6z\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.927950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.927329    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.928268 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.926814    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.928375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.926946    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:38.928448 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:38.928418    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-fcb5e5b64832: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:16:38.928448 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:38.928438    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:16:38.928569 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:38.928562    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials podName:5cbcb57b-39d7-46f0-810e-95fdb7f644d3 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:39.428481235 +0000 UTC m=+2365.920297761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh" (UID: "5cbcb57b-39d7-46f0-810e-95fdb7f644d3") : secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:16:38.947957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:38.947677    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kx6z\" (UniqueName: \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-kube-api-access-6kx6z\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:39.386959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:39.386917    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"]
Oct 30 21:16:39.387545 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:39.387516    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh" podUID="5cbcb57b-39d7-46f0-810e-95fdb7f644d3"
Oct 30 21:16:39.436211 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:39.436180    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:39.436385 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:39.436335    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-fcb5e5b64832: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:16:39.436385 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:39.436354    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:16:39.436463 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:39.436407    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials podName:5cbcb57b-39d7-46f0-810e-95fdb7f644d3 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:40.436392123 +0000 UTC m=+2366.928208663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh" (UID: "5cbcb57b-39d7-46f0-810e-95fdb7f644d3") : secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:16:40.215042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.215001    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw" podUID="7058cb33-50b0-42ae-a5ba-9ded28af6cc7" containerName="payload-processing" containerID="cri-o://253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b" gracePeriod=25
Oct 30 21:16:40.316766 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.316736    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:40.325376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.325341    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:40.349940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.349912    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-routing-overlay-volume\") pod \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") "
Oct 30 21:16:40.350112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.350002    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-plugins-config-volume\") pod \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") "
Oct 30 21:16:40.350232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.350112    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-kube-api-access-6kx6z\" (UniqueName: \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-kube-api-access-6kx6z\") pod \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") "
Oct 30 21:16:40.350908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.350887    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-routing-overlay-volume" pod "5cbcb57b-39d7-46f0-810e-95fdb7f644d3" (UID: "5cbcb57b-39d7-46f0-810e-95fdb7f644d3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:40.351315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.351252    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-plugins-config-volume" pod "5cbcb57b-39d7-46f0-810e-95fdb7f644d3" (UID: "5cbcb57b-39d7-46f0-810e-95fdb7f644d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:40.353271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.353243    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-kube-api-access-6kx6z" pod "5cbcb57b-39d7-46f0-810e-95fdb7f644d3" (UID: "5cbcb57b-39d7-46f0-810e-95fdb7f644d3"). InnerVolumeSpecName "kube-api-access-6kx6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:40.386256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.386199    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c" podUID="67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c" containerName="payload-processing" containerID="cri-o://1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01" gracePeriod=25
Oct 30 21:16:40.452213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.452177    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh\" (UID: \"5cbcb57b-39d7-46f0-810e-95fdb7f644d3\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:40.452396 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:40.452353    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-fcb5e5b64832: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:16:40.452396 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:40.452378    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh: secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:16:40.452495 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:40.452455    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials podName:5cbcb57b-39d7-46f0-810e-95fdb7f644d3 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:42.452432477 +0000 UTC m=+2368.944249005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh" (UID: "5cbcb57b-39d7-46f0-810e-95fdb7f644d3") : secret "agc-cred-fcb5e5b64832" not found
Oct 30 21:16:40.452637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.452617    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kx6z\" (UniqueName: \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-kube-api-access-6kx6z\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:40.452744 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.452644    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:40.452744 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.452660    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:40.742679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.742649    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:16:40.776835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.776795    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-plugins-config-volume\") pod \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") "
Oct 30 21:16:40.777021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.776997    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-routing-overlay-volume\") pod \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") "
Oct 30 21:16:40.777099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.777083    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-provider-credentials\") pod \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") "
Oct 30 21:16:40.777200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.777187    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-kube-api-access-gll7j\" (UniqueName: \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-kube-api-access-gll7j\") pod \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\" (UID: \"7058cb33-50b0-42ae-a5ba-9ded28af6cc7\") "
Oct 30 21:16:40.779936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.779903    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-kube-api-access-gll7j" pod "7058cb33-50b0-42ae-a5ba-9ded28af6cc7" (UID: "7058cb33-50b0-42ae-a5ba-9ded28af6cc7"). InnerVolumeSpecName "kube-api-access-gll7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:40.780257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.780236    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-plugins-config-volume" pod "7058cb33-50b0-42ae-a5ba-9ded28af6cc7" (UID: "7058cb33-50b0-42ae-a5ba-9ded28af6cc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:40.780552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.780534    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-routing-overlay-volume" pod "7058cb33-50b0-42ae-a5ba-9ded28af6cc7" (UID: "7058cb33-50b0-42ae-a5ba-9ded28af6cc7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:40.782960 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.782935    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-provider-credentials" pod "7058cb33-50b0-42ae-a5ba-9ded28af6cc7" (UID: "7058cb33-50b0-42ae-a5ba-9ded28af6cc7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:40.891880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.880300    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:40.891880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.880342    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:40.891880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.880358    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:40.891880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:40.880373    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gll7j\" (UniqueName: \"kubernetes.io/projected/7058cb33-50b0-42ae-a5ba-9ded28af6cc7-kube-api-access-gll7j\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.094399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.094370    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:41.205879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.188247    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials\") pod \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") "
Oct 30 21:16:41.205879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.188319    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-plugins-config-volume\") pod \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") "
Oct 30 21:16:41.205879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.188562    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-kube-api-access-stbgk\" (UniqueName: \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-kube-api-access-stbgk\") pod \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") "
Oct 30 21:16:41.205879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.188848    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-routing-overlay-volume\") pod \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\" (UID: \"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c\") "
Oct 30 21:16:41.205879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.202827    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-plugins-config-volume" pod "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c" (UID: "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:41.206316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.206110    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-routing-overlay-volume" pod "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c" (UID: "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:41.208561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.208403    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-kube-api-access-stbgk" pod "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c" (UID: "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c"). InnerVolumeSpecName "kube-api-access-stbgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:41.208561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.208519    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials" pod "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c" (UID: "67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:41.247882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.244924    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w" podUID="50a2e74d-3e80-4667-a322-fa39e0b985ea" containerName="payload-processing" containerID="cri-o://7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961" gracePeriod=25
Oct 30 21:16:41.293537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.293501    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.293537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.293538    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.293748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.293553    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.293748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.293566    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stbgk\" (UniqueName: \"kubernetes.io/projected/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c-kube-api-access-stbgk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.331182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.331141    4810 generic.go:356] "Generic (PLEG): container finished" podID="7058cb33-50b0-42ae-a5ba-9ded28af6cc7" containerID="253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b" exitCode=0
Oct 30 21:16:41.331398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.331386    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"
Oct 30 21:16:41.332193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.331910    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw" event={"ID":"7058cb33-50b0-42ae-a5ba-9ded28af6cc7","Type":"ContainerDied","Data":"253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b"}
Oct 30 21:16:41.332193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.331967    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw" event={"ID":"7058cb33-50b0-42ae-a5ba-9ded28af6cc7","Type":"ContainerDied","Data":"909ff9037fc85d2c128a349124e8f822fcea9946db6acb978ca422acfc792865"}
Oct 30 21:16:41.332193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.331997    4810 scope.go:122] "RemoveContainer" containerID="253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b"
Oct 30 21:16:41.334187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.334160    4810 generic.go:356] "Generic (PLEG): container finished" podID="67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c" containerID="1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01" exitCode=0
Oct 30 21:16:41.334312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.334284    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"
Oct 30 21:16:41.334394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.334283    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c" event={"ID":"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c","Type":"ContainerDied","Data":"1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01"}
Oct 30 21:16:41.334394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.334370    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c" event={"ID":"67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c","Type":"ContainerDied","Data":"135e5a35d66de9be322f0c7bc2734906f100ec72d59884b82e0a4aca7c04c49a"}
Oct 30 21:16:41.334610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.334592    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"
Oct 30 21:16:41.366180 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.365850    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"]
Oct 30 21:16:41.372912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.372881    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-6b9dfbpjw"]
Oct 30 21:16:41.379876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.379829    4810 scope.go:122] "RemoveContainer" containerID="253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b"
Oct 30 21:16:41.380258 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:41.380200    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b\": container with ID starting with 253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b not found: ID does not exist" containerID="253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b"
Oct 30 21:16:41.381583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.380421    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b"} err="failed to get container status \"253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b\": rpc error: code = NotFound desc = could not find container \"253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b\": container with ID starting with 253eebae3089e877b7ebd833baf183e889931020810fc2c10db27da741828e7b not found: ID does not exist"
Oct 30 21:16:41.381583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.380564    4810 scope.go:122] "RemoveContainer" containerID="1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01"
Oct 30 21:16:41.409315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.409285    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"]
Oct 30 21:16:41.409472 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.409449    4810 scope.go:122] "RemoveContainer" containerID="1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01"
Oct 30 21:16:41.411468 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:41.411434    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01\": container with ID starting with 1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01 not found: ID does not exist" containerID="1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01"
Oct 30 21:16:41.411610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.411477    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01"} err="failed to get container status \"1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01\": rpc error: code = NotFound desc = could not find container \"1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01\": container with ID starting with 1d865693fc36d2a3e2c586c75a2bec6bd50edd013c55e149cd2c3ea60ad84f01 not found: ID does not exist"
Oct 30 21:16:41.424471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.424426    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d99fdjtnh"]
Oct 30 21:16:41.435263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.433742    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"]
Oct 30 21:16:41.441130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.441101    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-7bb65n55c"]
Oct 30 21:16:41.497962 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.496332    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5cbcb57b-39d7-46f0-810e-95fdb7f644d3-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.617980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.617723    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:41.674770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.674732    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbcb57b-39d7-46f0-810e-95fdb7f644d3" path="/var/lib/kubelet/pods/5cbcb57b-39d7-46f0-810e-95fdb7f644d3/volumes"
Oct 30 21:16:41.675397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.675368    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c" path="/var/lib/kubelet/pods/67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c/volumes"
Oct 30 21:16:41.676123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.676099    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7058cb33-50b0-42ae-a5ba-9ded28af6cc7" path="/var/lib/kubelet/pods/7058cb33-50b0-42ae-a5ba-9ded28af6cc7/volumes"
Oct 30 21:16:41.698988 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.698955    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-routing-overlay-volume\") pod \"50a2e74d-3e80-4667-a322-fa39e0b985ea\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") "
Oct 30 21:16:41.699141 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.699025    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-provider-credentials\" (UniqueName: \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-provider-credentials\") pod \"50a2e74d-3e80-4667-a322-fa39e0b985ea\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") "
Oct 30 21:16:41.699231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.699212    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-kube-api-access-xn24v\" (UniqueName: \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-kube-api-access-xn24v\") pod \"50a2e74d-3e80-4667-a322-fa39e0b985ea\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") "
Oct 30 21:16:41.699366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.699347    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-plugins-config-volume\") pod \"50a2e74d-3e80-4667-a322-fa39e0b985ea\" (UID: \"50a2e74d-3e80-4667-a322-fa39e0b985ea\") "
Oct 30 21:16:41.699481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.699455    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-routing-overlay-volume" pod "50a2e74d-3e80-4667-a322-fa39e0b985ea" (UID: "50a2e74d-3e80-4667-a322-fa39e0b985ea"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:41.699961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.699936    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.700086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.700032    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-plugins-config-volume" pod "50a2e74d-3e80-4667-a322-fa39e0b985ea" (UID: "50a2e74d-3e80-4667-a322-fa39e0b985ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:41.702517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.702486    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-kube-api-access-xn24v" pod "50a2e74d-3e80-4667-a322-fa39e0b985ea" (UID: "50a2e74d-3e80-4667-a322-fa39e0b985ea"). InnerVolumeSpecName "kube-api-access-xn24v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:41.721159 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.721126    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-provider-credentials" pod "50a2e74d-3e80-4667-a322-fa39e0b985ea" (UID: "50a2e74d-3e80-4667-a322-fa39e0b985ea"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:41.803156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.802899    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn24v\" (UniqueName: \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-kube-api-access-xn24v\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.803156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.802941    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a2e74d-3e80-4667-a322-fa39e0b985ea-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.803156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:41.802958    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50a2e74d-3e80-4667-a322-fa39e0b985ea-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.349318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.349146    4810 generic.go:356] "Generic (PLEG): container finished" podID="50a2e74d-3e80-4667-a322-fa39e0b985ea" containerID="7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961" exitCode=0
Oct 30 21:16:42.349318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.349185    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w" event={"ID":"50a2e74d-3e80-4667-a322-fa39e0b985ea","Type":"ContainerDied","Data":"7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961"}
Oct 30 21:16:42.349318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.349226    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"
Oct 30 21:16:42.349318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.349240    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w" event={"ID":"50a2e74d-3e80-4667-a322-fa39e0b985ea","Type":"ContainerDied","Data":"38cb2db17af0f4b6ec7edb3e36474207cf4dd570e2c6b6b8ecc5b5e8fb1dfcfc"}
Oct 30 21:16:42.349318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.349271    4810 scope.go:122] "RemoveContainer" containerID="7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961"
Oct 30 21:16:42.361578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.361558    4810 scope.go:122] "RemoveContainer" containerID="7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961"
Oct 30 21:16:42.361816 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:42.361797    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961\": container with ID starting with 7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961 not found: ID does not exist" containerID="7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961"
Oct 30 21:16:42.361880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.361825    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961"} err="failed to get container status \"7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961\": rpc error: code = NotFound desc = could not find container \"7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961\": container with ID starting with 7b81609a36eeef2cd05f2baad5584dae1908b25a96c93725da982731c9ce8961 not found: ID does not exist"
Oct 30 21:16:42.377745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.377718    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"]
Oct 30 21:16:42.385590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.385567    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6494c5b757-s2z9w"]
Oct 30 21:16:42.534075 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.533955    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm" podUID="80e6f191-eaf5-4d65-834a-10176982b9f7" containerName="payload-processing" containerID="cri-o://186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78" gracePeriod=25
Oct 30 21:16:42.561563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:42.561497    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh" podUID="53419711-85c4-4dc5-93e0-524f8ff26222" containerName="payload-processing" containerID="cri-o://dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea" gracePeriod=25
Oct 30 21:16:43.127739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.121913    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:16:43.127739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.124484    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:43.155885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.155205    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-plugins-config-volume\") pod \"53419711-85c4-4dc5-93e0-524f8ff26222\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") "
Oct 30 21:16:43.155885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.155347    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-provider-credentials\") pod \"80e6f191-eaf5-4d65-834a-10176982b9f7\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") "
Oct 30 21:16:43.155885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.155416    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-plugins-config-volume\") pod \"80e6f191-eaf5-4d65-834a-10176982b9f7\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") "
Oct 30 21:16:43.155885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.155462    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-routing-overlay-volume\") pod \"53419711-85c4-4dc5-93e0-524f8ff26222\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") "
Oct 30 21:16:43.155885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.155537    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-provider-credentials\" (UniqueName: \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-provider-credentials\") pod \"53419711-85c4-4dc5-93e0-524f8ff26222\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") "
Oct 30 21:16:43.156269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.156110    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-kube-api-access-v2nv4\" (UniqueName: \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-kube-api-access-v2nv4\") pod \"53419711-85c4-4dc5-93e0-524f8ff26222\" (UID: \"53419711-85c4-4dc5-93e0-524f8ff26222\") "
Oct 30 21:16:43.156269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.156170    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-kube-api-access-99xc9\" (UniqueName: \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-kube-api-access-99xc9\") pod \"80e6f191-eaf5-4d65-834a-10176982b9f7\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") "
Oct 30 21:16:43.156269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.156212    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-routing-overlay-volume\") pod \"80e6f191-eaf5-4d65-834a-10176982b9f7\" (UID: \"80e6f191-eaf5-4d65-834a-10176982b9f7\") "
Oct 30 21:16:43.157752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.157283    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-routing-overlay-volume" pod "53419711-85c4-4dc5-93e0-524f8ff26222" (UID: "53419711-85c4-4dc5-93e0-524f8ff26222"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:43.157752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.157677    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-routing-overlay-volume" pod "80e6f191-eaf5-4d65-834a-10176982b9f7" (UID: "80e6f191-eaf5-4d65-834a-10176982b9f7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:43.157752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.157706    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-plugins-config-volume" pod "53419711-85c4-4dc5-93e0-524f8ff26222" (UID: "53419711-85c4-4dc5-93e0-524f8ff26222"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:43.158269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.158235    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-plugins-config-volume" pod "80e6f191-eaf5-4d65-834a-10176982b9f7" (UID: "80e6f191-eaf5-4d65-834a-10176982b9f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:43.159899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.159657    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-kube-api-access-v2nv4" pod "53419711-85c4-4dc5-93e0-524f8ff26222" (UID: "53419711-85c4-4dc5-93e0-524f8ff26222"). InnerVolumeSpecName "kube-api-access-v2nv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:43.160144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.160116    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-kube-api-access-99xc9" pod "80e6f191-eaf5-4d65-834a-10176982b9f7" (UID: "80e6f191-eaf5-4d65-834a-10176982b9f7"). InnerVolumeSpecName "kube-api-access-99xc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:43.161837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.161801    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-provider-credentials" pod "80e6f191-eaf5-4d65-834a-10176982b9f7" (UID: "80e6f191-eaf5-4d65-834a-10176982b9f7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:43.162392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.162368    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-provider-credentials" pod "53419711-85c4-4dc5-93e0-524f8ff26222" (UID: "53419711-85c4-4dc5-93e0-524f8ff26222"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:43.259223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.259189    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:43.259223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.259219    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:43.259422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.259230    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:43.259422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.259239    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:43.259422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.259250    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2nv4\" (UniqueName: \"kubernetes.io/projected/53419711-85c4-4dc5-93e0-524f8ff26222-kube-api-access-v2nv4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:43.259422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.259260    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xc9\" (UniqueName: \"kubernetes.io/projected/80e6f191-eaf5-4d65-834a-10176982b9f7-kube-api-access-99xc9\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:43.259422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.259269    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80e6f191-eaf5-4d65-834a-10176982b9f7-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:43.259422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.259278    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53419711-85c4-4dc5-93e0-524f8ff26222-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:43.362938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.362899    4810 generic.go:356] "Generic (PLEG): container finished" podID="53419711-85c4-4dc5-93e0-524f8ff26222" containerID="dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea" exitCode=0
Oct 30 21:16:43.363402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.362947    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"
Oct 30 21:16:43.363402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.362968    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh" event={"ID":"53419711-85c4-4dc5-93e0-524f8ff26222","Type":"ContainerDied","Data":"dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea"}
Oct 30 21:16:43.363402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.363022    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh" event={"ID":"53419711-85c4-4dc5-93e0-524f8ff26222","Type":"ContainerDied","Data":"77689f9dfa33d5005ad90b9f6118bf3d37741edf29e57100530add1d302ffdb7"}
Oct 30 21:16:43.363402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.363047    4810 scope.go:122] "RemoveContainer" containerID="dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea"
Oct 30 21:16:43.366085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.366060    4810 generic.go:356] "Generic (PLEG): container finished" podID="80e6f191-eaf5-4d65-834a-10176982b9f7" containerID="186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78" exitCode=0
Oct 30 21:16:43.366224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.366098    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm" event={"ID":"80e6f191-eaf5-4d65-834a-10176982b9f7","Type":"ContainerDied","Data":"186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78"}
Oct 30 21:16:43.366224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.366122    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"
Oct 30 21:16:43.366224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.366137    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm" event={"ID":"80e6f191-eaf5-4d65-834a-10176982b9f7","Type":"ContainerDied","Data":"67c1517885f5bf7970879484b7ba795d0a5e292e76a3534e22943851ee7b2949"}
Oct 30 21:16:43.429910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.425031    4810 scope.go:122] "RemoveContainer" containerID="dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea"
Oct 30 21:16:43.429910 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:43.425514    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea\": container with ID starting with dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea not found: ID does not exist" containerID="dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea"
Oct 30 21:16:43.429910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.425561    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea"} err="failed to get container status \"dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea\": rpc error: code = NotFound desc = could not find container \"dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea\": container with ID starting with dde8d8d53b20b8bbb8bcd7201bc4f463c9edbb2ddba2889b5ccbf86ad76873ea not found: ID does not exist"
Oct 30 21:16:43.429910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.425586    4810 scope.go:122] "RemoveContainer" containerID="186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78"
Oct 30 21:16:43.461320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.461276    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"]
Oct 30 21:16:43.467962 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.467914    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-d5989vvmh"]
Oct 30 21:16:43.469904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.469880    4810 scope.go:122] "RemoveContainer" containerID="186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78"
Oct 30 21:16:43.470242 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:43.470220    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78\": container with ID starting with 186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78 not found: ID does not exist" containerID="186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78"
Oct 30 21:16:43.470359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.470251    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78"} err="failed to get container status \"186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78\": rpc error: code = NotFound desc = could not find container \"186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78\": container with ID starting with 186b0e6a2ed153d7e1e8b2ccd306d0ed747ba74d86b221c75a0865ae4c41ef78 not found: ID does not exist"
Oct 30 21:16:43.487878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.483940    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"]
Oct 30 21:16:43.504782 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.504753    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-f495qwpjm"]
Oct 30 21:16:43.666897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.666811    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a2e74d-3e80-4667-a322-fa39e0b985ea" path="/var/lib/kubelet/pods/50a2e74d-3e80-4667-a322-fa39e0b985ea/volumes"
Oct 30 21:16:43.667236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.667221    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53419711-85c4-4dc5-93e0-524f8ff26222" path="/var/lib/kubelet/pods/53419711-85c4-4dc5-93e0-524f8ff26222/volumes"
Oct 30 21:16:43.667611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:43.667597    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e6f191-eaf5-4d65-834a-10176982b9f7" path="/var/lib/kubelet/pods/80e6f191-eaf5-4d65-834a-10176982b9f7/volumes"
Oct 30 21:16:47.257692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:47.257656    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:16:47.440118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:47.440087    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"]
Oct 30 21:16:52.441882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.440934    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr" podUID="16213cf3-133d-44f0-b17e-83d102d07a36" containerName="payload-processing" containerID="cri-o://b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917" gracePeriod=25
Oct 30 21:16:52.787236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.787205    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:52.927280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.927245    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-provider-credentials\" (UniqueName: \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-provider-credentials\") pod \"16213cf3-133d-44f0-b17e-83d102d07a36\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") "
Oct 30 21:16:52.927472 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.927411    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-plugins-config-volume\") pod \"16213cf3-133d-44f0-b17e-83d102d07a36\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") "
Oct 30 21:16:52.927607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.927595    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-routing-overlay-volume\") pod \"16213cf3-133d-44f0-b17e-83d102d07a36\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") "
Oct 30 21:16:52.927656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.927617    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-kube-api-access-25l66\" (UniqueName: \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-kube-api-access-25l66\") pod \"16213cf3-133d-44f0-b17e-83d102d07a36\" (UID: \"16213cf3-133d-44f0-b17e-83d102d07a36\") "
Oct 30 21:16:52.927812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.927787    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-plugins-config-volume" pod "16213cf3-133d-44f0-b17e-83d102d07a36" (UID: "16213cf3-133d-44f0-b17e-83d102d07a36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:52.928045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.928021    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-routing-overlay-volume" pod "16213cf3-133d-44f0-b17e-83d102d07a36" (UID: "16213cf3-133d-44f0-b17e-83d102d07a36"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:52.928400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.928383    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:52.928463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.928403    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16213cf3-133d-44f0-b17e-83d102d07a36-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:52.929778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.929477    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-provider-credentials" pod "16213cf3-133d-44f0-b17e-83d102d07a36" (UID: "16213cf3-133d-44f0-b17e-83d102d07a36"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:52.929898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:52.929876    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-kube-api-access-25l66" pod "16213cf3-133d-44f0-b17e-83d102d07a36" (UID: "16213cf3-133d-44f0-b17e-83d102d07a36"). InnerVolumeSpecName "kube-api-access-25l66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:53.034944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.033083    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25l66\" (UniqueName: \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-kube-api-access-25l66\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:53.034944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.033121    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16213cf3-133d-44f0-b17e-83d102d07a36-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:53.089977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.089935    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"]
Oct 30 21:16:53.142883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.091358    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7058cb33-50b0-42ae-a5ba-9ded28af6cc7" containerName="payload-processing"
Oct 30 21:16:53.156916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.143114    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7058cb33-50b0-42ae-a5ba-9ded28af6cc7" containerName="payload-processing"
Oct 30 21:16:53.157244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.157216    4810 cpu_manager.go:414] "containerMap: removing container" podUID="50a2e74d-3e80-4667-a322-fa39e0b985ea" containerName="payload-processing"
Oct 30 21:16:53.162881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.157345    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a2e74d-3e80-4667-a322-fa39e0b985ea" containerName="payload-processing"
Oct 30 21:16:53.163111 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.163087    4810 cpu_manager.go:414] "containerMap: removing container" podUID="67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c" containerName="payload-processing"
Oct 30 21:16:53.171247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.171209    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c" containerName="payload-processing"
Oct 30 21:16:53.195882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.171463    4810 cpu_manager.go:414] "containerMap: removing container" podUID="53419711-85c4-4dc5-93e0-524f8ff26222" containerName="payload-processing"
Oct 30 21:16:53.196148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.196116    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53419711-85c4-4dc5-93e0-524f8ff26222" containerName="payload-processing"
Oct 30 21:16:53.197235 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.197210    4810 cpu_manager.go:414] "containerMap: removing container" podUID="16213cf3-133d-44f0-b17e-83d102d07a36" containerName="payload-processing"
Oct 30 21:16:53.197437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.197416    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16213cf3-133d-44f0-b17e-83d102d07a36" containerName="payload-processing"
Oct 30 21:16:53.197561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.197547    4810 cpu_manager.go:414] "containerMap: removing container" podUID="80e6f191-eaf5-4d65-834a-10176982b9f7" containerName="payload-processing"
Oct 30 21:16:53.197649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.197637    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e6f191-eaf5-4d65-834a-10176982b9f7" containerName="payload-processing"
Oct 30 21:16:53.208263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.208230    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a2e74d-3e80-4667-a322-fa39e0b985ea" containerName="payload-processing"
Oct 30 21:16:53.239640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.239605    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d9a5cf-731a-4ad4-a00c-39cf4b26ca3c" containerName="payload-processing"
Oct 30 21:16:53.249965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.249918    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="53419711-85c4-4dc5-93e0-524f8ff26222" containerName="payload-processing"
Oct 30 21:16:53.250205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.250188    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="16213cf3-133d-44f0-b17e-83d102d07a36" containerName="payload-processing"
Oct 30 21:16:53.250301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.250289    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e6f191-eaf5-4d65-834a-10176982b9f7" containerName="payload-processing"
Oct 30 21:16:53.250418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.250407    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7058cb33-50b0-42ae-a5ba-9ded28af6cc7" containerName="payload-processing"
Oct 30 21:16:53.271355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.271316    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"]
Oct 30 21:16:53.287056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.282483    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.313879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.308240    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e8063c\""
Oct 30 21:16:53.313879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.308910    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"openshift-service-ca.crt\""
Oct 30 21:16:53.313879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.308985    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"kube-root-ca.crt\""
Oct 30 21:16:53.313879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.309064    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"payload-processing-external-model-e2e--3ca9ce38-dockercfg-6fdks\""
Oct 30 21:16:53.313879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.309102    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"agc-cred-6735c3b85df0\""
Oct 30 21:16:53.313879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.309162    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"routing-overlay\""
Oct 30 21:16:53.344250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.342818    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.344250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.342898    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.344250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.343166    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj6bw\" (UniqueName: \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-kube-api-access-xj6bw\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.344250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.343226    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.444248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.444214    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj6bw\" (UniqueName: \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-kube-api-access-xj6bw\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.444739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.444268    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.444739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.444299    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.444739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.444321    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.445138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.445110    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.445285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.445262    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.447255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.447221    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.454339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.454314    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj6bw\" (UniqueName: \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-kube-api-access-xj6bw\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.515294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.515261    4810 generic.go:356] "Generic (PLEG): container finished" podID="16213cf3-133d-44f0-b17e-83d102d07a36" containerID="b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917" exitCode=0
Oct 30 21:16:53.515475 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.515322    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"
Oct 30 21:16:53.515475 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.515348    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr" event={"ID":"16213cf3-133d-44f0-b17e-83d102d07a36","Type":"ContainerDied","Data":"b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917"}
Oct 30 21:16:53.515475 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.515430    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr" event={"ID":"16213cf3-133d-44f0-b17e-83d102d07a36","Type":"ContainerDied","Data":"2e8e8b56e33c1f20bfb244dd397c67af5fcbc6cc903317a42b417e105b16f080"}
Oct 30 21:16:53.515475 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.515457    4810 scope.go:122] "RemoveContainer" containerID="b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917"
Oct 30 21:16:53.526463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.526445    4810 scope.go:122] "RemoveContainer" containerID="b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917"
Oct 30 21:16:53.526755 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:53.526735    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917\": container with ID starting with b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917 not found: ID does not exist" containerID="b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917"
Oct 30 21:16:53.526819 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.526766    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917"} err="failed to get container status \"b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917\": rpc error: code = NotFound desc = could not find container \"b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917\": container with ID starting with b5e71f195663d1f646b3af6e00fdb859d230cceabe70966416fee39f01324917 not found: ID does not exist"
Oct 30 21:16:53.543170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.543107    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"]
Oct 30 21:16:53.551028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.551005    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59cd9876d5-km8cr"]
Oct 30 21:16:53.640356 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.640327    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:53.665070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.665046    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16213cf3-133d-44f0-b17e-83d102d07a36" path="/var/lib/kubelet/pods/16213cf3-133d-44f0-b17e-83d102d07a36/volumes"
Oct 30 21:16:53.931359 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:53.931315    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6d44b5_e17b_4a4a_bfb5_46e9df1e95bf.slice/crio-99116375123bfb8c67cc5cd73e696b3b3232a80cc27f1e77ff4d6c6fa18ff1f2 WatchSource:0}: Error finding container 99116375123bfb8c67cc5cd73e696b3b3232a80cc27f1e77ff4d6c6fa18ff1f2: Status 404 returned error can't find the container with id 99116375123bfb8c67cc5cd73e696b3b3232a80cc27f1e77ff4d6c6fa18ff1f2
Oct 30 21:16:53.931831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:53.931802    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"]
Oct 30 21:16:54.530070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.530030    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw" event={"ID":"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf","Type":"ContainerStarted","Data":"6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511"}
Oct 30 21:16:54.530619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.530096    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw" event={"ID":"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf","Type":"ContainerStarted","Data":"99116375123bfb8c67cc5cd73e696b3b3232a80cc27f1e77ff4d6c6fa18ff1f2"}
Oct 30 21:16:54.566481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.566422    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw" podStartSLOduration=2.566404253 podStartE2EDuration="2.566404253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:54.560411832 +0000 UTC m=+2381.052228369" watchObservedRunningTime="2026-10-30 21:16:54.566404253 +0000 UTC m=+2381.058220803"
Oct 30 21:16:54.903101 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.903065    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"]
Oct 30 21:16:54.908222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.908198    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:54.919353 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.919322    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"]
Oct 30 21:16:54.974759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.974722    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:54.974958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.974781    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:54.975051 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.974995    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:54.975127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.975048    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:54.975127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.975102    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39578b95-1622-484d-b298-366283d7c78d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:54.975269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.975236    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:54.975457 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.975281    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwbqj\" (UniqueName: \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-kube-api-access-lwbqj\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:54.975457 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.975321    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:54.975621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:54.975459    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39578b95-1622-484d-b298-366283d7c78d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.077253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077219    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39578b95-1622-484d-b298-366283d7c78d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.077253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077259    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.077534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077280    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.077534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077321    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.077534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077356    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.077534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077397    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39578b95-1622-484d-b298-366283d7c78d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.077534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077508    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.077814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077535    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwbqj\" (UniqueName: \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-kube-api-access-lwbqj\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.077814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077569    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.077814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077773    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.078015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077807    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.078015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.077994    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.078131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.078114    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39578b95-1622-484d-b298-366283d7c78d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.078424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.078402    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.080100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.080072    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.080384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.080360    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39578b95-1622-484d-b298-366283d7c78d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.087482 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.087457    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.087830 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.087811    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwbqj\" (UniqueName: \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-kube-api-access-lwbqj\") pod \"maas-default-gateway-openshift-default-77f556d647-9cqch\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.157167 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:16:55.157096    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:16:55.227399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.227352    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:55.515445 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:55.515412    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39578b95_1622_484d_b298_366283d7c78d.slice/crio-21df41ef463393d6014b9e3b0290e085bd5c29fb86510866aeb7156d95c7098b WatchSource:0}: Error finding container 21df41ef463393d6014b9e3b0290e085bd5c29fb86510866aeb7156d95c7098b: Status 404 returned error can't find the container with id 21df41ef463393d6014b9e3b0290e085bd5c29fb86510866aeb7156d95c7098b
Oct 30 21:16:55.516170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.516130    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"]
Oct 30 21:16:55.518848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.518814    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:16:55.518957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.518926    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:16:55.519039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.518971    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:16:55.542444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.542415    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" event={"ID":"39578b95-1622-484d-b298-366283d7c78d","Type":"ContainerStarted","Data":"21df41ef463393d6014b9e3b0290e085bd5c29fb86510866aeb7156d95c7098b"}
Oct 30 21:16:55.542776 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:55.542688    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:16:56.553607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.553572    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" event={"ID":"39578b95-1622-484d-b298-366283d7c78d","Type":"ContainerStarted","Data":"d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb"}
Oct 30 21:16:56.658878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.658130    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" podStartSLOduration=2.6581085570000003 podStartE2EDuration="2.658108557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:56.600364402 +0000 UTC m=+2383.092180948" watchObservedRunningTime="2026-10-30 21:16:56.658108557 +0000 UTC m=+2383.149925106"
Oct 30 21:16:56.693884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.690638    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"]
Oct 30 21:16:56.729880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.726825    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"]
Oct 30 21:16:56.729880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.726965    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.748882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.746631    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99668e\""
Oct 30 21:16:56.748882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.747401    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"routing-overlay\""
Oct 30 21:16:56.748882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.747502    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"payload-processing-external-model-e2e--789fdc40-dockercfg-swfd6\""
Oct 30 21:16:56.748882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.747556    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"agc-cred-25fd5cbcab4f\""
Oct 30 21:16:56.748882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.747626    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"kube-root-ca.crt\""
Oct 30 21:16:56.748882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.747693    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"openshift-service-ca.crt\""
Oct 30 21:16:56.855911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.855802    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r4kq\" (UniqueName: \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-kube-api-access-7r4kq\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.856332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.856304    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.856585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.856567    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.856743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.856728    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.964878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.960037    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.964878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.960198    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.964878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.960260    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.964878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.960344    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r4kq\" (UniqueName: \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-kube-api-access-7r4kq\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.964878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.962712    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.964878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.963461    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:56.982924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:56.974081    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:57.062820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:57.060684    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r4kq\" (UniqueName: \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-kube-api-access-7r4kq\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:57.134931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:57.130131    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:57.231918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:57.229170    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:16:57.235878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:57.234064    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-9cqch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 21:16:57.235878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:57.234121    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 21:16:57.548837 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:16:57.548804    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefeecd51_da58_4b3d_b956_569ffd6e868f.slice/crio-3a3e6144f58d04ad63f11d0f3b2a18bc797baa1f22c3e5bce62ea161ef8f1a90 WatchSource:0}: Error finding container 3a3e6144f58d04ad63f11d0f3b2a18bc797baa1f22c3e5bce62ea161ef8f1a90: Status 404 returned error can't find the container with id 3a3e6144f58d04ad63f11d0f3b2a18bc797baa1f22c3e5bce62ea161ef8f1a90
Oct 30 21:16:57.549360 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:57.549334    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"]
Oct 30 21:16:57.563964 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:57.563934    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg" event={"ID":"efeecd51-da58-4b3d-b956-569ffd6e868f","Type":"ContainerStarted","Data":"3a3e6144f58d04ad63f11d0f3b2a18bc797baa1f22c3e5bce62ea161ef8f1a90"}
Oct 30 21:16:58.227838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:58.227800    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-9cqch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 21:16:58.228063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:58.227873    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 21:16:58.587585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:58.587482    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg" event={"ID":"efeecd51-da58-4b3d-b956-569ffd6e868f","Type":"ContainerStarted","Data":"2fe22f96e236bd2d835d061fa2db35f38ef1e4e4565239cdff767f68fd469a2c"}
Oct 30 21:16:58.588191 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:58.588170    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:16:58.610602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:58.610543    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg" podStartSLOduration=2.6105280950000003 podStartE2EDuration="2.610528095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:58.606201453 +0000 UTC m=+2385.098017999" watchObservedRunningTime="2026-10-30 21:16:58.610528095 +0000 UTC m=+2385.102344641"
Oct 30 21:16:59.227763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:59.227726    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-9cqch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 21:16:59.227986 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:16:59.227782    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 21:17:00.228629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:00.228594    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-9cqch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 21:17:00.229127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:00.228652    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 21:17:01.228237 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:01.228198    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-9cqch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 21:17:01.228429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:01.228252    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 21:17:02.227785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:02.227742    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-9cqch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 21:17:02.228280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:02.227822    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 21:17:03.228476 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:03.228437    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-9cqch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body=
Oct 30 21:17:03.228912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:03.228506    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused"
Oct 30 21:17:04.409679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:04.409643    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:17:04.411231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:04.411198    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:17:04.412354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:04.412217    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:17:06.557794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:06.557766    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:17:09.612600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:09.612537    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:17:12.109487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:12.109453    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:17:14.665785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:14.665754    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:17:14.737881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:14.737347    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:17:17.536890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:17.536845    4810 scope.go:122] "RemoveContainer" containerID="6181b29cf7acbcb4be7b32818baa582521d8127a9b785eb14e36ca5265b04a73"
Oct 30 21:17:20.027527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:20.027489    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"]
Oct 30 21:17:20.033203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:20.033175    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3f8913-kmxrc"]
Oct 30 21:17:21.664645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:21.664605    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d19158-93cf-4b69-bf7c-46f7d3d849dd" path="/var/lib/kubelet/pods/65d19158-93cf-4b69-bf7c-46f7d3d849dd/volumes"
Oct 30 21:17:25.157990 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:17:25.157955    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:17:46.032228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:46.032190    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"]
Oct 30 21:17:46.038277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:46.038246    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fe7dae-4pw8c"]
Oct 30 21:17:47.664103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:47.664075    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93319499-4c40-4c24-a456-47a018b1164e" path="/var/lib/kubelet/pods/93319499-4c40-4c24-a456-47a018b1164e/volumes"
Oct 30 21:17:49.738429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:49.738389    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"]
Oct 30 21:17:49.739313 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:49.739274    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy" containerID="cri-o://f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30" gracePeriod=30
Oct 30 21:17:54.388901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:54.388819    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:17:55.056783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.056750    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 21:17:55.160767 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:17:55.160731    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:17:55.170437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.170402    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-istio-token\" (UniqueName: \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-istio-token\") pod \"20525970-334f-4fe4-984a-1f6c89329c3f\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") "
Oct 30 21:17:55.170584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.170517    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-envoy\") pod \"20525970-334f-4fe4-984a-1f6c89329c3f\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") "
Oct 30 21:17:55.170630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.170596    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-kube-api-access-j6zl9\" (UniqueName: \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-kube-api-access-j6zl9\") pod \"20525970-334f-4fe4-984a-1f6c89329c3f\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") "
Oct 30 21:17:55.170667 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.170654    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-certs\") pod \"20525970-334f-4fe4-984a-1f6c89329c3f\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") "
Oct 30 21:17:55.170722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.170704    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-credential-socket\") pod \"20525970-334f-4fe4-984a-1f6c89329c3f\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") "
Oct 30 21:17:55.172026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.170828    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20525970-334f-4fe4-984a-1f6c89329c3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20525970-334f-4fe4-984a-1f6c89329c3f-istiod-ca-cert\") pod \"20525970-334f-4fe4-984a-1f6c89329c3f\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") "
Oct 30 21:17:55.172026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.171002    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20525970-334f-4fe4-984a-1f6c89329c3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20525970-334f-4fe4-984a-1f6c89329c3f-istio-podinfo\") pod \"20525970-334f-4fe4-984a-1f6c89329c3f\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") "
Oct 30 21:17:55.172026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.171039    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-data\") pod \"20525970-334f-4fe4-984a-1f6c89329c3f\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") "
Oct 30 21:17:55.172026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.171196    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-socket\") pod \"20525970-334f-4fe4-984a-1f6c89329c3f\" (UID: \"20525970-334f-4fe4-984a-1f6c89329c3f\") "
Oct 30 21:17:55.172026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.171486    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-credential-socket" pod "20525970-334f-4fe4-984a-1f6c89329c3f" (UID: "20525970-334f-4fe4-984a-1f6c89329c3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:55.172026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.171535    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20525970-334f-4fe4-984a-1f6c89329c3f-istiod-ca-cert" pod "20525970-334f-4fe4-984a-1f6c89329c3f" (UID: "20525970-334f-4fe4-984a-1f6c89329c3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:55.172026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.171712    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-data" pod "20525970-334f-4fe4-984a-1f6c89329c3f" (UID: "20525970-334f-4fe4-984a-1f6c89329c3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:55.172026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.171712    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-certs" pod "20525970-334f-4fe4-984a-1f6c89329c3f" (UID: "20525970-334f-4fe4-984a-1f6c89329c3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:55.172026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.171975    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-socket" pod "20525970-334f-4fe4-984a-1f6c89329c3f" (UID: "20525970-334f-4fe4-984a-1f6c89329c3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:55.172623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.172510    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:55.172623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.172535    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:55.172623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.172550    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20525970-334f-4fe4-984a-1f6c89329c3f-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:55.172623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.172565    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:55.172623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.172578    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:55.173163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.173139    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-istio-token" pod "20525970-334f-4fe4-984a-1f6c89329c3f" (UID: "20525970-334f-4fe4-984a-1f6c89329c3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:55.173651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.173629    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-kube-api-access-j6zl9" pod "20525970-334f-4fe4-984a-1f6c89329c3f" (UID: "20525970-334f-4fe4-984a-1f6c89329c3f"). InnerVolumeSpecName "kube-api-access-j6zl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:55.175076 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.175053    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20525970-334f-4fe4-984a-1f6c89329c3f-istio-podinfo" pod "20525970-334f-4fe4-984a-1f6c89329c3f" (UID: "20525970-334f-4fe4-984a-1f6c89329c3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:17:55.175183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.175078    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-envoy" pod "20525970-334f-4fe4-984a-1f6c89329c3f" (UID: "20525970-334f-4fe4-984a-1f6c89329c3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:55.239298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.237989    4810 generic.go:356] "Generic (PLEG): container finished" podID="20525970-334f-4fe4-984a-1f6c89329c3f" containerID="f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30" exitCode=0
Oct 30 21:17:55.239298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.238490    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" event={"ID":"20525970-334f-4fe4-984a-1f6c89329c3f","Type":"ContainerDied","Data":"f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30"}
Oct 30 21:17:55.239298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.238617    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"
Oct 30 21:17:55.239298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.238628    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" event={"ID":"20525970-334f-4fe4-984a-1f6c89329c3f","Type":"ContainerDied","Data":"11022623c495ae0e7c6336d399c79d53d47aca5e24ce7b01fbae2d025caa5185"}
Oct 30 21:17:55.239298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.238661    4810 scope.go:122] "RemoveContainer" containerID="f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30"
Oct 30 21:17:55.266250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.266225    4810 scope.go:122] "RemoveContainer" containerID="f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30"
Oct 30 21:17:55.266665 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:17:55.266627    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30\": container with ID starting with f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30 not found: ID does not exist" containerID="f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30"
Oct 30 21:17:55.266792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.266675    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30"} err="failed to get container status \"f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30\": rpc error: code = NotFound desc = could not find container \"f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30\": container with ID starting with f07b592b8875b6edb8c181fb59894e1f6c68ddb4c7fddc2344d1ea7d79709a30 not found: ID does not exist"
Oct 30 21:17:55.274495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.274108    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20525970-334f-4fe4-984a-1f6c89329c3f-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:55.274495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.274141    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:55.274495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.274156    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20525970-334f-4fe4-984a-1f6c89329c3f-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:55.274495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.274171    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6zl9\" (UniqueName: \"kubernetes.io/projected/20525970-334f-4fe4-984a-1f6c89329c3f-kube-api-access-j6zl9\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:55.276556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.275599    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"]
Oct 30 21:17:55.280411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.280390    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln"]
Oct 30 21:17:55.712073 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:55.703999    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" path="/var/lib/kubelet/pods/20525970-334f-4fe4-984a-1f6c89329c3f/volumes"
Oct 30 21:17:56.026452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:56.026352    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:56.026628 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:56.026448    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-hm2ln" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:56.787573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:56.786677    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"]
Oct 30 21:17:56.787573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:56.786965    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy" containerID="cri-o://2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d" gracePeriod=30
Oct 30 21:17:58.043881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:58.043317    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"]
Oct 30 21:17:58.075889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:58.071323    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf115403-d57mv"]
Oct 30 21:17:59.669168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:17:59.669133    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9896c083-00b2-425b-a9c4-14f79bd66cac" path="/var/lib/kubelet/pods/9896c083-00b2-425b-a9c4-14f79bd66cac/volumes"
Oct 30 21:18:00.236421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.236386    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"]
Oct 30 21:18:00.237208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.237192    4810 cpu_manager.go:414] "containerMap: removing container" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy"
Oct 30 21:18:00.237272 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.237209    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy"
Oct 30 21:18:00.237332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.237323    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="20525970-334f-4fe4-984a-1f6c89329c3f" containerName="istio-proxy"
Oct 30 21:18:00.244879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.244818    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.253145 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.253115    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"openshift-service-ca.crt\""
Oct 30 21:18:00.253287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.253152    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e07442\""
Oct 30 21:18:00.253287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.253182    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"kube-root-ca.crt\""
Oct 30 21:18:00.253411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.253298    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"routing-overlay\""
Oct 30 21:18:00.253411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.253398    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"agc-cred-20646cd4a159\""
Oct 30 21:18:00.253517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.253415    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"payload-processing-external-model-e2e--4fdc1812-dockercfg-mmmpb\""
Oct 30 21:18:00.260123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.260099    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"]
Oct 30 21:18:00.277305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.277280    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.277500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.277479    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqh6p\" (UniqueName: \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-kube-api-access-jqh6p\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.277586 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.277528    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.277633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.277609    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.417988 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.382900    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.418339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.384780    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.418535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.418509    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqh6p\" (UniqueName: \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-kube-api-access-jqh6p\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.418682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.418667    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.418795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.418779    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.443923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.431967    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.465887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.452647    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqh6p\" (UniqueName: \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-kube-api-access-jqh6p\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.491920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.471143    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.572145 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.572108    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:00.875214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:00.875181    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"]
Oct 30 21:18:00.878033 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:18:00.878004    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0278d2bb_4e87_47aa_8daa_935f1bc41ed0.slice/crio-608d886a60a0eeaf333e137bb070a4aa720d0aacfa63d8db1daaffd817ce2766 WatchSource:0}: Error finding container 608d886a60a0eeaf333e137bb070a4aa720d0aacfa63d8db1daaffd817ce2766: Status 404 returned error can't find the container with id 608d886a60a0eeaf333e137bb070a4aa720d0aacfa63d8db1daaffd817ce2766
Oct 30 21:18:01.503591 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:01.503552    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj" event={"ID":"0278d2bb-4e87-47aa-8daa-935f1bc41ed0","Type":"ContainerStarted","Data":"372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9"}
Oct 30 21:18:01.503829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:01.503613    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj" event={"ID":"0278d2bb-4e87-47aa-8daa-935f1bc41ed0","Type":"ContainerStarted","Data":"608d886a60a0eeaf333e137bb070a4aa720d0aacfa63d8db1daaffd817ce2766"}
Oct 30 21:18:01.503829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:01.503636    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:01.520199 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:01.520149    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj" podStartSLOduration=1.520136189 podStartE2EDuration="1.520136189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:01.51934176 +0000 UTC m=+2448.011158308" watchObservedRunningTime="2026-10-30 21:18:01.520136189 +0000 UTC m=+2448.011952734"
Oct 30 21:18:02.059674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.059645    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 21:18:02.163780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.163743    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-certs\") pod \"34d300d6-6375-45bb-a33b-1f66ca279494\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") "
Oct 30 21:18:02.164040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.163820    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34d300d6-6375-45bb-a33b-1f66ca279494-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d300d6-6375-45bb-a33b-1f66ca279494-istio-podinfo\") pod \"34d300d6-6375-45bb-a33b-1f66ca279494\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") "
Oct 30 21:18:02.164040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.163891    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-socket\") pod \"34d300d6-6375-45bb-a33b-1f66ca279494\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") "
Oct 30 21:18:02.164040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.163939    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-credential-socket\") pod \"34d300d6-6375-45bb-a33b-1f66ca279494\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") "
Oct 30 21:18:02.164232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.164042    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-istio-token\" (UniqueName: \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-istio-token\") pod \"34d300d6-6375-45bb-a33b-1f66ca279494\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") "
Oct 30 21:18:02.164232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.164075    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-data\") pod \"34d300d6-6375-45bb-a33b-1f66ca279494\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") "
Oct 30 21:18:02.164232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.164085    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-certs" pod "34d300d6-6375-45bb-a33b-1f66ca279494" (UID: "34d300d6-6375-45bb-a33b-1f66ca279494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:02.164232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.164174    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-kube-api-access-wg8nq\" (UniqueName: \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-kube-api-access-wg8nq\") pod \"34d300d6-6375-45bb-a33b-1f66ca279494\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") "
Oct 30 21:18:02.164435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.164238    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-credential-socket" pod "34d300d6-6375-45bb-a33b-1f66ca279494" (UID: "34d300d6-6375-45bb-a33b-1f66ca279494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:02.164435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.164247    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-socket" pod "34d300d6-6375-45bb-a33b-1f66ca279494" (UID: "34d300d6-6375-45bb-a33b-1f66ca279494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:02.164435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.164298    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d300d6-6375-45bb-a33b-1f66ca279494-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d300d6-6375-45bb-a33b-1f66ca279494-istiod-ca-cert\") pod \"34d300d6-6375-45bb-a33b-1f66ca279494\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") "
Oct 30 21:18:02.164435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.164361    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-envoy\") pod \"34d300d6-6375-45bb-a33b-1f66ca279494\" (UID: \"34d300d6-6375-45bb-a33b-1f66ca279494\") "
Oct 30 21:18:02.164435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.164393    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-data" pod "34d300d6-6375-45bb-a33b-1f66ca279494" (UID: "34d300d6-6375-45bb-a33b-1f66ca279494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:02.164739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.164677    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d300d6-6375-45bb-a33b-1f66ca279494-istiod-ca-cert" pod "34d300d6-6375-45bb-a33b-1f66ca279494" (UID: "34d300d6-6375-45bb-a33b-1f66ca279494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:02.165123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.165095    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:02.165123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.165124    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:02.165325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.165139    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:02.165325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.165154    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:02.165325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.165169    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d300d6-6375-45bb-a33b-1f66ca279494-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:02.166213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.166185    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34d300d6-6375-45bb-a33b-1f66ca279494-istio-podinfo" pod "34d300d6-6375-45bb-a33b-1f66ca279494" (UID: "34d300d6-6375-45bb-a33b-1f66ca279494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:18:02.166596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.166572    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-kube-api-access-wg8nq" pod "34d300d6-6375-45bb-a33b-1f66ca279494" (UID: "34d300d6-6375-45bb-a33b-1f66ca279494"). InnerVolumeSpecName "kube-api-access-wg8nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:02.167162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.167138    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-istio-token" pod "34d300d6-6375-45bb-a33b-1f66ca279494" (UID: "34d300d6-6375-45bb-a33b-1f66ca279494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:02.167349 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.167329    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-envoy" pod "34d300d6-6375-45bb-a33b-1f66ca279494" (UID: "34d300d6-6375-45bb-a33b-1f66ca279494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:02.266563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.266523    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:02.266563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.266557    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg8nq\" (UniqueName: \"kubernetes.io/projected/34d300d6-6375-45bb-a33b-1f66ca279494-kube-api-access-wg8nq\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:02.266563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.266569    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d300d6-6375-45bb-a33b-1f66ca279494-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:02.266822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.266577    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d300d6-6375-45bb-a33b-1f66ca279494-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:02.517984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.517889    4810 generic.go:356] "Generic (PLEG): container finished" podID="34d300d6-6375-45bb-a33b-1f66ca279494" containerID="2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d" exitCode=0
Oct 30 21:18:02.518149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.517998    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"
Oct 30 21:18:02.518149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.518033    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" event={"ID":"34d300d6-6375-45bb-a33b-1f66ca279494","Type":"ContainerDied","Data":"2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d"}
Oct 30 21:18:02.518149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.518100    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2" event={"ID":"34d300d6-6375-45bb-a33b-1f66ca279494","Type":"ContainerDied","Data":"44251a6a68701fdabd803b2173251ce0faf0196d0a49f5cdb93900248c184c26"}
Oct 30 21:18:02.518328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.518180    4810 scope.go:122] "RemoveContainer" containerID="2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d"
Oct 30 21:18:02.530916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.530900    4810 scope.go:122] "RemoveContainer" containerID="2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d"
Oct 30 21:18:02.531179 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:02.531160    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d\": container with ID starting with 2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d not found: ID does not exist" containerID="2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d"
Oct 30 21:18:02.531237 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.531187    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d"} err="failed to get container status \"2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d\": rpc error: code = NotFound desc = could not find container \"2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d\": container with ID starting with 2b588eb97ef3b4acc652e29ef156d750882576472c1140a9b94f6fb9a560aa3d not found: ID does not exist"
Oct 30 21:18:02.544413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.544385    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"]
Oct 30 21:18:02.549708 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:02.549683    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-wzld2"]
Oct 30 21:18:03.680343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:03.680307    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" path="/var/lib/kubelet/pods/34d300d6-6375-45bb-a33b-1f66ca279494/volumes"
Oct 30 21:18:07.759457 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:07.759422    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"]
Oct 30 21:18:07.760011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:07.759654    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerName="istio-proxy" containerID="cri-o://b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d" gracePeriod=30
Oct 30 21:18:12.522367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:12.522326    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:18:13.090992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.090962    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:18:13.258042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258005    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-socket\") pod \"07c2eacd-60e4-4cf2-a16d-734cbf585086\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") "
Oct 30 21:18:13.258250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258094    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-credential-socket\") pod \"07c2eacd-60e4-4cf2-a16d-734cbf585086\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") "
Oct 30 21:18:13.258250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258114    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-envoy\") pod \"07c2eacd-60e4-4cf2-a16d-734cbf585086\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") "
Oct 30 21:18:13.258250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258193    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c2eacd-60e4-4cf2-a16d-734cbf585086-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c2eacd-60e4-4cf2-a16d-734cbf585086-istiod-ca-cert\") pod \"07c2eacd-60e4-4cf2-a16d-734cbf585086\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") "
Oct 30 21:18:13.258250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258232    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-token\" (UniqueName: \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-token\") pod \"07c2eacd-60e4-4cf2-a16d-734cbf585086\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") "
Oct 30 21:18:13.258483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258279    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-data\") pod \"07c2eacd-60e4-4cf2-a16d-734cbf585086\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") "
Oct 30 21:18:13.258483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258358    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-certs\") pod \"07c2eacd-60e4-4cf2-a16d-734cbf585086\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") "
Oct 30 21:18:13.258483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258360    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-socket" pod "07c2eacd-60e4-4cf2-a16d-734cbf585086" (UID: "07c2eacd-60e4-4cf2-a16d-734cbf585086"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:13.258483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258395    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-kube-api-access-6flq2\" (UniqueName: \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-kube-api-access-6flq2\") pod \"07c2eacd-60e4-4cf2-a16d-734cbf585086\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") "
Oct 30 21:18:13.258483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258463    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-credential-socket" pod "07c2eacd-60e4-4cf2-a16d-734cbf585086" (UID: "07c2eacd-60e4-4cf2-a16d-734cbf585086"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:13.258483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258477    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-podinfo\") pod \"07c2eacd-60e4-4cf2-a16d-734cbf585086\" (UID: \"07c2eacd-60e4-4cf2-a16d-734cbf585086\") "
Oct 30 21:18:13.258795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258660    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-data" pod "07c2eacd-60e4-4cf2-a16d-734cbf585086" (UID: "07c2eacd-60e4-4cf2-a16d-734cbf585086"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:13.258795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258729    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c2eacd-60e4-4cf2-a16d-734cbf585086-istiod-ca-cert" pod "07c2eacd-60e4-4cf2-a16d-734cbf585086" (UID: "07c2eacd-60e4-4cf2-a16d-734cbf585086"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:13.258942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.258843    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-certs" pod "07c2eacd-60e4-4cf2-a16d-734cbf585086" (UID: "07c2eacd-60e4-4cf2-a16d-734cbf585086"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:13.259206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.259186    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07c2eacd-60e4-4cf2-a16d-734cbf585086-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:13.259206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.259206    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:13.259374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.259215    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:13.259374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.259224    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:13.259374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.259233    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:13.260509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.260485    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-envoy" pod "07c2eacd-60e4-4cf2-a16d-734cbf585086" (UID: "07c2eacd-60e4-4cf2-a16d-734cbf585086"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:13.260737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.260711    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-podinfo" pod "07c2eacd-60e4-4cf2-a16d-734cbf585086" (UID: "07c2eacd-60e4-4cf2-a16d-734cbf585086"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:18:13.261023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.261006    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-kube-api-access-6flq2" pod "07c2eacd-60e4-4cf2-a16d-734cbf585086" (UID: "07c2eacd-60e4-4cf2-a16d-734cbf585086"). InnerVolumeSpecName "kube-api-access-6flq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:13.261077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.261032    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-token" pod "07c2eacd-60e4-4cf2-a16d-734cbf585086" (UID: "07c2eacd-60e4-4cf2-a16d-734cbf585086"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:13.360783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.360721    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:13.360783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.360748    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6flq2\" (UniqueName: \"kubernetes.io/projected/07c2eacd-60e4-4cf2-a16d-734cbf585086-kube-api-access-6flq2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:13.360783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.360759    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:13.360783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.360768    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07c2eacd-60e4-4cf2-a16d-734cbf585086-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:13.640838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.640807    4810 generic.go:356] "Generic (PLEG): container finished" podID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerID="b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d" exitCode=0
Oct 30 21:18:13.641276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.640875    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"
Oct 30 21:18:13.641276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.640894    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" event={"ID":"07c2eacd-60e4-4cf2-a16d-734cbf585086","Type":"ContainerDied","Data":"b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d"}
Oct 30 21:18:13.641276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.640951    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz" event={"ID":"07c2eacd-60e4-4cf2-a16d-734cbf585086","Type":"ContainerDied","Data":"a2fdb1e42387e818e570d18f1d055832834949507e302dc7d90a27625e9dd257"}
Oct 30 21:18:13.641276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.640973    4810 scope.go:122] "RemoveContainer" containerID="b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d"
Oct 30 21:18:13.654222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.654205    4810 scope.go:122] "RemoveContainer" containerID="b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d"
Oct 30 21:18:13.654495 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:13.654475    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d\": container with ID starting with b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d not found: ID does not exist" containerID="b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d"
Oct 30 21:18:13.654569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.654504    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d"} err="failed to get container status \"b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d\": rpc error: code = NotFound desc = could not find container \"b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d\": container with ID starting with b93a763b96771419d40b95b6f7d74eeaa21427a9a5d7eaa43844b4d3a01e448d not found: ID does not exist"
Oct 30 21:18:13.679847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.679816    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"]
Oct 30 21:18:13.686341 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:13.686248    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-fx4qz"]
Oct 30 21:18:15.663637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:15.663604    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" path="/var/lib/kubelet/pods/07c2eacd-60e4-4cf2-a16d-734cbf585086/volumes"
Oct 30 21:18:17.639658 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:17.639254    4810 scope.go:122] "RemoveContainer" containerID="9974e7b993d1f8bd43e83edb3e3a4feff0a8ede56f41e29ea5e43ee56737331b"
Oct 30 21:18:23.042609 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:23.042570    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"]
Oct 30 21:18:23.052678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:23.052643    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae547c11-h7fls"]
Oct 30 21:18:23.688968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:23.687891    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056785f6-bd60-4661-8a82-9143a5ef7487" path="/var/lib/kubelet/pods/056785f6-bd60-4661-8a82-9143a5ef7487/volumes"
Oct 30 21:18:25.157347 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:25.157258    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:18:26.111075 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.111039    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"]
Oct 30 21:18:26.111921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.111901    4810 cpu_manager.go:414] "containerMap: removing container" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerName="istio-proxy"
Oct 30 21:18:26.111984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.111923    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerName="istio-proxy"
Oct 30 21:18:26.111984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.111974    4810 cpu_manager.go:414] "containerMap: removing container" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy"
Oct 30 21:18:26.111984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.111979    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy"
Oct 30 21:18:26.112103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.112098    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c2eacd-60e4-4cf2-a16d-734cbf585086" containerName="istio-proxy"
Oct 30 21:18:26.112145 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.112115    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d300d6-6375-45bb-a33b-1f66ca279494" containerName="istio-proxy"
Oct 30 21:18:26.118253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.117876    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.122780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.122759    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:26.122971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.122845    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:18:26.124891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.124867    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"]
Oct 30 21:18:26.297193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.297147    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn64j\" (UniqueName: \"kubernetes.io/projected/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-kube-api-access-dn64j\") pod \"maas-api-revoke-sub-69e22cc6-j5sfv\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.297580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.297283    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e22cc6-j5sfv\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.297580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.297460    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e22cc6-j5sfv\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.398328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.398297    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn64j\" (UniqueName: \"kubernetes.io/projected/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-kube-api-access-dn64j\") pod \"maas-api-revoke-sub-69e22cc6-j5sfv\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.398487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.398356    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e22cc6-j5sfv\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.398487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.398462    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e22cc6-j5sfv\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.399111 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.399091    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e22cc6-j5sfv\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.400784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.400763    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e22cc6-j5sfv\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.406971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.406941    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn64j\" (UniqueName: \"kubernetes.io/projected/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-kube-api-access-dn64j\") pod \"maas-api-revoke-sub-69e22cc6-j5sfv\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.439790 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.439763    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:26.731058 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.731028    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"]
Oct 30 21:18:26.731192 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:18:26.731077    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f499fe9_1e19_443a_a38a_e40f00ed5dc1.slice/crio-8d85edd553d5051bf74a33c1bf775f633976a77a5345adf6fbc3d0396f006faa WatchSource:0}: Error finding container 8d85edd553d5051bf74a33c1bf775f633976a77a5345adf6fbc3d0396f006faa: Status 404 returned error can't find the container with id 8d85edd553d5051bf74a33c1bf775f633976a77a5345adf6fbc3d0396f006faa
Oct 30 21:18:26.765736 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:26.765708    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv" event={"ID":"3f499fe9-1e19-443a-a38a-e40f00ed5dc1","Type":"ContainerStarted","Data":"8d85edd553d5051bf74a33c1bf775f633976a77a5345adf6fbc3d0396f006faa"}
Oct 30 21:18:27.777171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:27.777137    4810 generic.go:356] "Generic (PLEG): container finished" podID="3f499fe9-1e19-443a-a38a-e40f00ed5dc1" containerID="fba7fefd5295d7ec8beffdf11b141585864369106764232e23624a0a898c683c" exitCode=0
Oct 30 21:18:27.777631 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:27.777220    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv" event={"ID":"3f499fe9-1e19-443a-a38a-e40f00ed5dc1","Type":"ContainerDied","Data":"fba7fefd5295d7ec8beffdf11b141585864369106764232e23624a0a898c683c"}
Oct 30 21:18:28.966765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:28.966739    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:29.048231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.048153    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-kube-api-access-dn64j\" (UniqueName: \"kubernetes.io/projected/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-kube-api-access-dn64j\") pod \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") "
Oct 30 21:18:29.048231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.048220    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-openshift-ca\") pod \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") "
Oct 30 21:18:29.048416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.048386    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-xks-ca\") pod \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\" (UID: \"3f499fe9-1e19-443a-a38a-e40f00ed5dc1\") "
Oct 30 21:18:29.048593 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.048566    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-openshift-ca" pod "3f499fe9-1e19-443a-a38a-e40f00ed5dc1" (UID: "3f499fe9-1e19-443a-a38a-e40f00ed5dc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:29.049113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.049086    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:29.050491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.050464    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-kube-api-access-dn64j" pod "3f499fe9-1e19-443a-a38a-e40f00ed5dc1" (UID: "3f499fe9-1e19-443a-a38a-e40f00ed5dc1"). InnerVolumeSpecName "kube-api-access-dn64j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:29.050602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.050556    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-xks-ca" pod "3f499fe9-1e19-443a-a38a-e40f00ed5dc1" (UID: "3f499fe9-1e19-443a-a38a-e40f00ed5dc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:29.150566 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.150533    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn64j\" (UniqueName: \"kubernetes.io/projected/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-kube-api-access-dn64j\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:29.150566 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.150563    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f499fe9-1e19-443a-a38a-e40f00ed5dc1-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:29.797053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.797006    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"
Oct 30 21:18:29.797053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.797040    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv" event={"ID":"3f499fe9-1e19-443a-a38a-e40f00ed5dc1","Type":"ContainerDied","Data":"8d85edd553d5051bf74a33c1bf775f633976a77a5345adf6fbc3d0396f006faa"}
Oct 30 21:18:29.797309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:29.797083    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d85edd553d5051bf74a33c1bf775f633976a77a5345adf6fbc3d0396f006faa"
Oct 30 21:18:34.036126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:34.036084    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"]
Oct 30 21:18:34.042602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:34.042568    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17241f60-cstqt"]
Oct 30 21:18:35.663757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:35.663724    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9292ee4b-2272-41ad-afcb-3757e6f93df4" path="/var/lib/kubelet/pods/9292ee4b-2272-41ad-afcb-3757e6f93df4/volumes"
Oct 30 21:18:37.116634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:37.116584    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"]
Oct 30 21:18:37.790801 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:37.790764    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:18:42.118041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.117998    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw" podUID="bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf" containerName="payload-processing" containerID="cri-o://6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511" gracePeriod=25
Oct 30 21:18:42.403791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.403748    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:18:42.503345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.503316    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-plugins-config-volume\") pod \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") "
Oct 30 21:18:42.503532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.503357    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-routing-overlay-volume\") pod \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") "
Oct 30 21:18:42.503532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.503497    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-kube-api-access-xj6bw\" (UniqueName: \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-kube-api-access-xj6bw\") pod \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") "
Oct 30 21:18:42.503629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.503617    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-provider-credentials\") pod \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\" (UID: \"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf\") "
Oct 30 21:18:42.503749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.503725    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-routing-overlay-volume" pod "bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf" (UID: "bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:42.503834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.503784    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-plugins-config-volume" pod "bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf" (UID: "bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:42.504319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.504294    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.504319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.504320    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.505882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.505843    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-provider-credentials" pod "bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf" (UID: "bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:42.505973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.505893    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-kube-api-access-xj6bw" pod "bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf" (UID: "bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf"). InnerVolumeSpecName "kube-api-access-xj6bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:42.606321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.606294    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj6bw\" (UniqueName: \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-kube-api-access-xj6bw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.606321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.606319    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.912688 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.912649    4810 generic.go:356] "Generic (PLEG): container finished" podID="bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf" containerID="6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511" exitCode=0
Oct 30 21:18:42.912903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.912694    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"
Oct 30 21:18:42.912903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.912735    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw" event={"ID":"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf","Type":"ContainerDied","Data":"6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511"}
Oct 30 21:18:42.912903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.912792    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw" event={"ID":"bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf","Type":"ContainerDied","Data":"99116375123bfb8c67cc5cd73e696b3b3232a80cc27f1e77ff4d6c6fa18ff1f2"}
Oct 30 21:18:42.912903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.912816    4810 scope.go:122] "RemoveContainer" containerID="6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511"
Oct 30 21:18:42.926240 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.926216    4810 scope.go:122] "RemoveContainer" containerID="6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511"
Oct 30 21:18:42.926546 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:42.926523    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511\": container with ID starting with 6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511 not found: ID does not exist" containerID="6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511"
Oct 30 21:18:42.926656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.926553    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511"} err="failed to get container status \"6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511\": rpc error: code = NotFound desc = could not find container \"6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511\": container with ID starting with 6ec8a3ecfa5f7d5520866a7e932d86051620682597965d1804075b01f3c31511 not found: ID does not exist"
Oct 30 21:18:42.941522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.941486    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"]
Oct 30 21:18:42.945976 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:42.945946    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-57cbldpdw"]
Oct 30 21:18:43.665926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:43.665894    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf" path="/var/lib/kubelet/pods/bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf/volumes"
Oct 30 21:18:45.029666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:45.029635    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"]
Oct 30 21:18:45.034518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:45.034485    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a633ccfb-4lb5j"]
Oct 30 21:18:45.663789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:45.663753    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7158dcac-f804-4aaa-92bb-e6a83f5056b2" path="/var/lib/kubelet/pods/7158dcac-f804-4aaa-92bb-e6a83f5056b2/volumes"
Oct 30 21:18:52.372120 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.372077    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"]
Oct 30 21:18:52.372973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.372954    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3f499fe9-1e19-443a-a38a-e40f00ed5dc1" containerName="revoke-keys"
Oct 30 21:18:52.372973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.372972    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f499fe9-1e19-443a-a38a-e40f00ed5dc1" containerName="revoke-keys"
Oct 30 21:18:52.373134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.372988    4810 cpu_manager.go:414] "containerMap: removing container" podUID="bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf" containerName="payload-processing"
Oct 30 21:18:52.373134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.372993    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf" containerName="payload-processing"
Oct 30 21:18:52.373243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.373145    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f499fe9-1e19-443a-a38a-e40f00ed5dc1" containerName="revoke-keys"
Oct 30 21:18:52.373243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.373156    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6d44b5-e17b-4a4a-bfb5-46e9df1e95bf" containerName="payload-processing"
Oct 30 21:18:52.378707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.378682    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.386550 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.386519    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"]
Oct 30 21:18:52.386689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.386662    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0100150b71ea\""
Oct 30 21:18:52.541636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.541598    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-routing-overlay-volume\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.541806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.541759    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-plugins-config-volume\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.541851 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.541825    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9bgb\" (UniqueName: \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-kube-api-access-w9bgb\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.541971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.541956    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.642773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.642740    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-plugins-config-volume\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.642985 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.642925    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9bgb\" (UniqueName: \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-kube-api-access-w9bgb\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.643101 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.643079    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.643212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.643194    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-routing-overlay-volume\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.643614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.643593    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-plugins-config-volume\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.643713 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.643687    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-routing-overlay-volume\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.645880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.645844    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.651644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.651621    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9bgb\" (UniqueName: \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-kube-api-access-w9bgb\") pod \"payload-processing-external-model-f7847d4fb-xsn8q\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") " pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:52.711198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:52.711153    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:53.164891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:53.163320    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"]
Oct 30 21:18:53.225392 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:18:53.225355    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44766fb7_f2d9_4caa_8ba5_267678e48b4e.slice/crio-ebefb16c91d27038c68e60afccf9fec041ad2ee22c5a8ce110747b42e83b631c WatchSource:0}: Error finding container ebefb16c91d27038c68e60afccf9fec041ad2ee22c5a8ce110747b42e83b631c: Status 404 returned error can't find the container with id ebefb16c91d27038c68e60afccf9fec041ad2ee22c5a8ce110747b42e83b631c
Oct 30 21:18:53.602887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:53.602795    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"]
Oct 30 21:18:54.158558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:54.158519    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q" event={"ID":"44766fb7-f2d9-4caa-8ba5-267678e48b4e","Type":"ContainerStarted","Data":"4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab"}
Oct 30 21:18:54.158750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:54.158593    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:18:54.158750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:54.158612    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q" event={"ID":"44766fb7-f2d9-4caa-8ba5-267678e48b4e","Type":"ContainerStarted","Data":"ebefb16c91d27038c68e60afccf9fec041ad2ee22c5a8ce110747b42e83b631c"}
Oct 30 21:18:54.199336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:54.199269    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q" podStartSLOduration=2.199246347 podStartE2EDuration="2.199246347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:54.190070569 +0000 UTC m=+2500.681887119" watchObservedRunningTime="2026-10-30 21:18:54.199246347 +0000 UTC m=+2500.691062897"
Oct 30 21:18:54.244512 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:54.244480    4810 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0100150b71ea: secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:54.244512 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:54.244523    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q: secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:54.244785 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:54.244594    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials podName:44766fb7-f2d9-4caa-8ba5-267678e48b4e nodeName:}" failed. No retries permitted until 2026-10-30 21:18:54.744573992 +0000 UTC m=+2501.236390535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials") pod "payload-processing-external-model-f7847d4fb-xsn8q" (UID: "44766fb7-f2d9-4caa-8ba5-267678e48b4e") : secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:54.751190 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:54.751155    4810 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0100150b71ea: secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:54.751190 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:54.751196    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q: secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:54.751654 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:54.751248    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials podName:44766fb7-f2d9-4caa-8ba5-267678e48b4e nodeName:}" failed. No retries permitted until 2026-10-30 21:18:55.751229861 +0000 UTC m=+2502.243046399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials") pod "payload-processing-external-model-f7847d4fb-xsn8q" (UID: "44766fb7-f2d9-4caa-8ba5-267678e48b4e") : secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:55.162623 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:55.162581    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:18:55.769849 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:55.769818    4810 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0100150b71ea: secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:55.769849 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:55.769853    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q: secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:55.770297 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:55.769919    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials podName:44766fb7-f2d9-4caa-8ba5-267678e48b4e nodeName:}" failed. No retries permitted until 2026-10-30 21:18:57.769903565 +0000 UTC m=+2504.261720089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials") pod "payload-processing-external-model-f7847d4fb-xsn8q" (UID: "44766fb7-f2d9-4caa-8ba5-267678e48b4e") : secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:57.817847 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:57.817810    4810 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0100150b71ea: secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:57.817847 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:57.817853    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q: secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:57.818443 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:18:57.817961    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials podName:44766fb7-f2d9-4caa-8ba5-267678e48b4e nodeName:}" failed. No retries permitted until 2026-10-30 21:19:01.817942169 +0000 UTC m=+2508.309758699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials") pod "payload-processing-external-model-f7847d4fb-xsn8q" (UID: "44766fb7-f2d9-4caa-8ba5-267678e48b4e") : secret "agc-cred-0100150b71ea" not found
Oct 30 21:18:58.063886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.054389    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"]
Oct 30 21:18:58.063886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.063204    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e48c045-z97rc"]
Oct 30 21:18:58.694747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.694710    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"]
Oct 30 21:18:58.701840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.701813    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.708003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.707974    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"]
Oct 30 21:18:58.715352 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.715326    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"kube-root-ca.crt\""
Oct 30 21:18:58.716230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.715984    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"payload-processing-external-model-e2e--3ca9ce38-dockercfg-llrmb\""
Oct 30 21:18:58.716230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.716050    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"agc-cred-6b853fe6f75b\""
Oct 30 21:18:58.716230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.716120    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e8063c\""
Oct 30 21:18:58.716230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.716177    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"openshift-service-ca.crt\""
Oct 30 21:18:58.716498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.716242    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"routing-overlay\""
Oct 30 21:18:58.785882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.762025    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.786264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.786231    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.888033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.887989    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.889052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.888964    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgfk8\" (UniqueName: \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-kube-api-access-cgfk8\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.991711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.991594    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.991711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.991649    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.991987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.991801    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.991987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.991885    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgfk8\" (UniqueName: \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-kube-api-access-cgfk8\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.992534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.992501    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.993094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.993071    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:58.994644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:58.994613    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:59.000560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.000536    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgfk8\" (UniqueName: \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-kube-api-access-cgfk8\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:59.025444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.025412    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:18:59.235374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.235339    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"]
Oct 30 21:18:59.254332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.254154    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.256870 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.255308    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"]
Oct 30 21:18:59.265347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.264140    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"openshift-service-ca.crt\""
Oct 30 21:18:59.265347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.264833    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e1e0f5\""
Oct 30 21:18:59.265347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.264970    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"payload-processing-external-model-e2e--d2a8fa5c-dockercfg-p2str\""
Oct 30 21:18:59.265347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.265055    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"routing-overlay\""
Oct 30 21:18:59.265347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.265131    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"kube-root-ca.crt\""
Oct 30 21:18:59.265347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.265201    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"agc-cred-fcc812463930\""
Oct 30 21:18:59.422737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.422691    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.423008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.422745    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkdw4\" (UniqueName: \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-kube-api-access-mkdw4\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.423008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.422825    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.423162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.423008    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.457565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.457529    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"]
Oct 30 21:18:59.464316 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:18:59.464266    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1797ffd5_a4e6_4064_a217_76c29d658340.slice/crio-5cd1da2466ec63273a7dc0414d7af8928466b3550caec39d4390664334dd2101 WatchSource:0}: Error finding container 5cd1da2466ec63273a7dc0414d7af8928466b3550caec39d4390664334dd2101: Status 404 returned error can't find the container with id 5cd1da2466ec63273a7dc0414d7af8928466b3550caec39d4390664334dd2101
Oct 30 21:18:59.525397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.525360    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.525565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.525548    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.525618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.525579    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkdw4\" (UniqueName: \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-kube-api-access-mkdw4\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.525678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.525664    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.527040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.526977    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.527565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.527545    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.529973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.529935    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.538214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.538187    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkdw4\" (UniqueName: \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-kube-api-access-mkdw4\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.605100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.605062    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:18:59.674742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.674711    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1739f7-ed64-45a4-bddf-955945c8d139" path="/var/lib/kubelet/pods/4b1739f7-ed64-45a4-bddf-955945c8d139/volumes"
Oct 30 21:18:59.898522 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:18:59.898480    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadf8d51_ac66_44d4_9428_e17018c70940.slice/crio-de5ff3790bdc314462b9b82b561f260a045df1f7607d523eba221f25f0999ef6 WatchSource:0}: Error finding container de5ff3790bdc314462b9b82b561f260a045df1f7607d523eba221f25f0999ef6: Status 404 returned error can't find the container with id de5ff3790bdc314462b9b82b561f260a045df1f7607d523eba221f25f0999ef6
Oct 30 21:18:59.901124 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:18:59.901096    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"]
Oct 30 21:19:00.169997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.169908    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q" podUID="44766fb7-f2d9-4caa-8ba5-267678e48b4e" containerName="payload-processing" containerID="cri-o://4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab" gracePeriod=25
Oct 30 21:19:00.244642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.244602    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx" event={"ID":"1797ffd5-a4e6-4064-a217-76c29d658340","Type":"ContainerStarted","Data":"ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58"}
Oct 30 21:19:00.244642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.244662    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx" event={"ID":"1797ffd5-a4e6-4064-a217-76c29d658340","Type":"ContainerStarted","Data":"5cd1da2466ec63273a7dc0414d7af8928466b3550caec39d4390664334dd2101"}
Oct 30 21:19:00.244992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.244699    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:19:00.246556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.246529    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd" event={"ID":"cadf8d51-ac66-44d4-9428-e17018c70940","Type":"ContainerStarted","Data":"a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe"}
Oct 30 21:19:00.246680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.246570    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd" event={"ID":"cadf8d51-ac66-44d4-9428-e17018c70940","Type":"ContainerStarted","Data":"de5ff3790bdc314462b9b82b561f260a045df1f7607d523eba221f25f0999ef6"}
Oct 30 21:19:00.246680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.246591    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:19:00.264367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.264314    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx" podStartSLOduration=2.26429949 podStartE2EDuration="2.26429949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:00.261518773 +0000 UTC m=+2506.753335318" watchObservedRunningTime="2026-10-30 21:19:00.26429949 +0000 UTC m=+2506.756116035"
Oct 30 21:19:00.281901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.281825    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd" podStartSLOduration=1.2818085639999999 podStartE2EDuration="1.281808564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:00.279033141 +0000 UTC m=+2506.770849695" watchObservedRunningTime="2026-10-30 21:19:00.281808564 +0000 UTC m=+2506.773625111"
Oct 30 21:19:00.563657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.563622    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:19:00.664144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.664110    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-kube-api-access-w9bgb\" (UniqueName: \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-kube-api-access-w9bgb\") pod \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") "
Oct 30 21:19:00.664301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.664239    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-routing-overlay-volume\") pod \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") "
Oct 30 21:19:00.664384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.664363    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials\") pod \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") "
Oct 30 21:19:00.664430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.664410    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-plugins-config-volume\") pod \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\" (UID: \"44766fb7-f2d9-4caa-8ba5-267678e48b4e\") "
Oct 30 21:19:00.664590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.664557    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-routing-overlay-volume" pod "44766fb7-f2d9-4caa-8ba5-267678e48b4e" (UID: "44766fb7-f2d9-4caa-8ba5-267678e48b4e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:00.664885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.664841    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-plugins-config-volume" pod "44766fb7-f2d9-4caa-8ba5-267678e48b4e" (UID: "44766fb7-f2d9-4caa-8ba5-267678e48b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:00.664971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.664955    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:00.665011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.664978    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44766fb7-f2d9-4caa-8ba5-267678e48b4e-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:00.666893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.666871    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials" pod "44766fb7-f2d9-4caa-8ba5-267678e48b4e" (UID: "44766fb7-f2d9-4caa-8ba5-267678e48b4e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:00.666968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.666906    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-kube-api-access-w9bgb" pod "44766fb7-f2d9-4caa-8ba5-267678e48b4e" (UID: "44766fb7-f2d9-4caa-8ba5-267678e48b4e"). InnerVolumeSpecName "kube-api-access-w9bgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:00.766140 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.766108    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:00.766140 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:00.766136    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9bgb\" (UniqueName: \"kubernetes.io/projected/44766fb7-f2d9-4caa-8ba5-267678e48b4e-kube-api-access-w9bgb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:01.267566 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:01.267531    4810 generic.go:356] "Generic (PLEG): container finished" podID="44766fb7-f2d9-4caa-8ba5-267678e48b4e" containerID="4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab" exitCode=0
Oct 30 21:19:01.268071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:01.267604    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"
Oct 30 21:19:01.268071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:01.267617    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q" event={"ID":"44766fb7-f2d9-4caa-8ba5-267678e48b4e","Type":"ContainerDied","Data":"4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab"}
Oct 30 21:19:01.268071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:01.267713    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q" event={"ID":"44766fb7-f2d9-4caa-8ba5-267678e48b4e","Type":"ContainerDied","Data":"ebefb16c91d27038c68e60afccf9fec041ad2ee22c5a8ce110747b42e83b631c"}
Oct 30 21:19:01.268071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:01.267741    4810 scope.go:122] "RemoveContainer" containerID="4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab"
Oct 30 21:19:01.283999 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:01.283966    4810 scope.go:122] "RemoveContainer" containerID="4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab"
Oct 30 21:19:01.284320 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:19:01.284295    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab\": container with ID starting with 4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab not found: ID does not exist" containerID="4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab"
Oct 30 21:19:01.284398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:01.284332    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab"} err="failed to get container status \"4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab\": rpc error: code = NotFound desc = could not find container \"4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab\": container with ID starting with 4b5d00ea734d9e8b9aa5ea6c41ca10802ccfd3d950e63602086f6d10b8de57ab not found: ID does not exist"
Oct 30 21:19:01.296787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:01.296752    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"]
Oct 30 21:19:01.302980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:01.302954    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f7847d4fb-xsn8q"]
Oct 30 21:19:01.691369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:01.687815    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44766fb7-f2d9-4caa-8ba5-267678e48b4e" path="/var/lib/kubelet/pods/44766fb7-f2d9-4caa-8ba5-267678e48b4e/volumes"
Oct 30 21:19:11.270923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:11.270849    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:19:11.271412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:11.271347    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:19:13.306659 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:13.306624    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:19:17.742877 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:17.742823    4810 scope.go:122] "RemoveContainer" containerID="63b56814691fa6d87f0ff3b7a06604441ee2f9ae2cb14f56d2cccddb93f46fb3"
Oct 30 21:19:17.754477 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:17.754452    4810 scope.go:122] "RemoveContainer" containerID="2a40e75d384774339bfd4377389b3fab2bc027d05fbfc408fc6f08cdf9fea246"
Oct 30 21:19:20.030334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.030295    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"]
Oct 30 21:19:20.031162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.031130    4810 cpu_manager.go:414] "containerMap: removing container" podUID="44766fb7-f2d9-4caa-8ba5-267678e48b4e" containerName="payload-processing"
Oct 30 21:19:20.031162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.031163    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44766fb7-f2d9-4caa-8ba5-267678e48b4e" containerName="payload-processing"
Oct 30 21:19:20.031342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.031332    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="44766fb7-f2d9-4caa-8ba5-267678e48b4e" containerName="payload-processing"
Oct 30 21:19:20.037177 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.037140    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.041739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.041717    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:20.042319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.042297    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:19:20.043438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.043406    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"]
Oct 30 21:19:20.115606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.115571    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d973926a-s7mbx\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.115810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.115659    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d973926a-s7mbx\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.115910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.115878    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grhq\" (UniqueName: \"kubernetes.io/projected/8a321c97-7975-4980-b689-df29c5f9e0c1-kube-api-access-6grhq\") pod \"maas-api-revoke-sub-d973926a-s7mbx\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.217196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.217156    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d973926a-s7mbx\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.217392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.217259    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d973926a-s7mbx\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.217392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.217371    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grhq\" (UniqueName: \"kubernetes.io/projected/8a321c97-7975-4980-b689-df29c5f9e0c1-kube-api-access-6grhq\") pod \"maas-api-revoke-sub-d973926a-s7mbx\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.217937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.217907    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d973926a-s7mbx\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.219957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.219933    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d973926a-s7mbx\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.230141 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.230116    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grhq\" (UniqueName: \"kubernetes.io/projected/8a321c97-7975-4980-b689-df29c5f9e0c1-kube-api-access-6grhq\") pod \"maas-api-revoke-sub-d973926a-s7mbx\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.355060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.354969    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:20.639165 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:20.639136    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"]
Oct 30 21:19:20.639688 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:19:20.639659    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a321c97_7975_4980_b689_df29c5f9e0c1.slice/crio-b4b19033a6038da05035701b87c902633a69b876da4b74166b4af041e62ff5d3 WatchSource:0}: Error finding container b4b19033a6038da05035701b87c902633a69b876da4b74166b4af041e62ff5d3: Status 404 returned error can't find the container with id b4b19033a6038da05035701b87c902633a69b876da4b74166b4af041e62ff5d3
Oct 30 21:19:21.535240 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:21.535195    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx" event={"ID":"8a321c97-7975-4980-b689-df29c5f9e0c1","Type":"ContainerStarted","Data":"f16f742d11571c7c9fd15132ada05bbf7dfc96167f5c07fec731904aebde52b2"}
Oct 30 21:19:21.535691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:21.535253    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx" event={"ID":"8a321c97-7975-4980-b689-df29c5f9e0c1","Type":"ContainerStarted","Data":"b4b19033a6038da05035701b87c902633a69b876da4b74166b4af041e62ff5d3"}
Oct 30 21:19:21.555992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:21.555941    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx" podStartSLOduration=2.555928706 podStartE2EDuration="2.555928706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:21.552206024 +0000 UTC m=+2528.044022580" watchObservedRunningTime="2026-10-30 21:19:21.555928706 +0000 UTC m=+2528.047745300"
Oct 30 21:19:22.545207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:22.545171    4810 generic.go:356] "Generic (PLEG): container finished" podID="8a321c97-7975-4980-b689-df29c5f9e0c1" containerID="f16f742d11571c7c9fd15132ada05bbf7dfc96167f5c07fec731904aebde52b2" exitCode=0
Oct 30 21:19:22.545635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:22.545247    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx" event={"ID":"8a321c97-7975-4980-b689-df29c5f9e0c1","Type":"ContainerDied","Data":"f16f742d11571c7c9fd15132ada05bbf7dfc96167f5c07fec731904aebde52b2"}
Oct 30 21:19:23.740068 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:23.740041    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:23.932615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:23.932588    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-openshift-ca\") pod \"8a321c97-7975-4980-b689-df29c5f9e0c1\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") "
Oct 30 21:19:23.932817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:23.932803    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a321c97-7975-4980-b689-df29c5f9e0c1-kube-api-access-6grhq\" (UniqueName: \"kubernetes.io/projected/8a321c97-7975-4980-b689-df29c5f9e0c1-kube-api-access-6grhq\") pod \"8a321c97-7975-4980-b689-df29c5f9e0c1\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") "
Oct 30 21:19:23.932916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:23.932842    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-xks-ca\") pod \"8a321c97-7975-4980-b689-df29c5f9e0c1\" (UID: \"8a321c97-7975-4980-b689-df29c5f9e0c1\") "
Oct 30 21:19:23.932994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:23.932966    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-openshift-ca" pod "8a321c97-7975-4980-b689-df29c5f9e0c1" (UID: "8a321c97-7975-4980-b689-df29c5f9e0c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:23.933364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:23.933341    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:23.934992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:23.934966    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-xks-ca" pod "8a321c97-7975-4980-b689-df29c5f9e0c1" (UID: "8a321c97-7975-4980-b689-df29c5f9e0c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:23.935086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:23.935059    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a321c97-7975-4980-b689-df29c5f9e0c1-kube-api-access-6grhq" pod "8a321c97-7975-4980-b689-df29c5f9e0c1" (UID: "8a321c97-7975-4980-b689-df29c5f9e0c1"). InnerVolumeSpecName "kube-api-access-6grhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:24.034382 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:24.034346    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6grhq\" (UniqueName: \"kubernetes.io/projected/8a321c97-7975-4980-b689-df29c5f9e0c1-kube-api-access-6grhq\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:24.034382 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:24.034381    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a321c97-7975-4980-b689-df29c5f9e0c1-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:24.562613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:24.562582    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"
Oct 30 21:19:24.562810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:24.562624    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx" event={"ID":"8a321c97-7975-4980-b689-df29c5f9e0c1","Type":"ContainerDied","Data":"b4b19033a6038da05035701b87c902633a69b876da4b74166b4af041e62ff5d3"}
Oct 30 21:19:24.562810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:24.562681    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b19033a6038da05035701b87c902633a69b876da4b74166b4af041e62ff5d3"
Oct 30 21:19:25.029400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:25.029363    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"]
Oct 30 21:19:25.033607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:25.033575    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc17e63-dlxpq"]
Oct 30 21:19:25.157758 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:19:25.157720    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:19:25.664308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:25.664273    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7330b2a2-e159-4d20-9d40-d8da51079d24" path="/var/lib/kubelet/pods/7330b2a2-e159-4d20-9d40-d8da51079d24/volumes"
Oct 30 21:19:31.001385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.001347    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"]
Oct 30 21:19:31.674335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.674298    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"]
Oct 30 21:19:31.675129 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.675111    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8a321c97-7975-4980-b689-df29c5f9e0c1" containerName="revoke-keys"
Oct 30 21:19:31.675187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.675131    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a321c97-7975-4980-b689-df29c5f9e0c1" containerName="revoke-keys"
Oct 30 21:19:31.675309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.675299    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a321c97-7975-4980-b689-df29c5f9e0c1" containerName="revoke-keys"
Oct 30 21:19:31.680562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.680542    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:31.685150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.685124    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:31.685286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.685223    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:19:31.688076 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.688052    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"]
Oct 30 21:19:31.801943 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.801903    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ef9cc37-bk4l8\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:31.802116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.801952    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6vm\" (UniqueName: \"kubernetes.io/projected/1f38a2d5-c514-4d0d-b714-5149065e6e4b-kube-api-access-9g6vm\") pod \"maas-api-revoke-sub-9ef9cc37-bk4l8\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:31.802338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.802316    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ef9cc37-bk4l8\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:31.903926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.903850    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ef9cc37-bk4l8\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:31.904116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.904018    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ef9cc37-bk4l8\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:31.904116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.904038    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6vm\" (UniqueName: \"kubernetes.io/projected/1f38a2d5-c514-4d0d-b714-5149065e6e4b-kube-api-access-9g6vm\") pod \"maas-api-revoke-sub-9ef9cc37-bk4l8\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:31.904665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.904643    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ef9cc37-bk4l8\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:31.906315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.906295    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ef9cc37-bk4l8\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:31.918414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.918357    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6vm\" (UniqueName: \"kubernetes.io/projected/1f38a2d5-c514-4d0d-b714-5149065e6e4b-kube-api-access-9g6vm\") pod \"maas-api-revoke-sub-9ef9cc37-bk4l8\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:32.000042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:31.999956    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:32.290261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:32.290228    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"]
Oct 30 21:19:32.290649 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:19:32.290240    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f38a2d5_c514_4d0d_b714_5149065e6e4b.slice/crio-b5ffb037eeb45c91bf2398dd49ec32fc8235566c14dc877946c83e7d3bda41c4 WatchSource:0}: Error finding container b5ffb037eeb45c91bf2398dd49ec32fc8235566c14dc877946c83e7d3bda41c4: Status 404 returned error can't find the container with id b5ffb037eeb45c91bf2398dd49ec32fc8235566c14dc877946c83e7d3bda41c4
Oct 30 21:19:32.662504 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:32.662470    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8" event={"ID":"1f38a2d5-c514-4d0d-b714-5149065e6e4b","Type":"ContainerStarted","Data":"b5ffb037eeb45c91bf2398dd49ec32fc8235566c14dc877946c83e7d3bda41c4"}
Oct 30 21:19:33.670974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:33.670936    4810 generic.go:356] "Generic (PLEG): container finished" podID="1f38a2d5-c514-4d0d-b714-5149065e6e4b" containerID="ef431eb5a73cbf885509b67dd30a6035a4627250b6ad80c0a8d3e3b84bc027f2" exitCode=0
Oct 30 21:19:33.671454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:33.671014    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8" event={"ID":"1f38a2d5-c514-4d0d-b714-5149065e6e4b","Type":"ContainerDied","Data":"ef431eb5a73cbf885509b67dd30a6035a4627250b6ad80c0a8d3e3b84bc027f2"}
Oct 30 21:19:34.855354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:34.855320    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:34.954040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:34.953966    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f38a2d5-c514-4d0d-b714-5149065e6e4b-kube-api-access-9g6vm\" (UniqueName: \"kubernetes.io/projected/1f38a2d5-c514-4d0d-b714-5149065e6e4b-kube-api-access-9g6vm\") pod \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") "
Oct 30 21:19:34.954197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:34.954107    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-xks-ca\") pod \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") "
Oct 30 21:19:34.954243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:34.954221    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-openshift-ca\") pod \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\" (UID: \"1f38a2d5-c514-4d0d-b714-5149065e6e4b\") "
Oct 30 21:19:34.954638 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:34.954611    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-openshift-ca" pod "1f38a2d5-c514-4d0d-b714-5149065e6e4b" (UID: "1f38a2d5-c514-4d0d-b714-5149065e6e4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:34.954852 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:34.954828    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:34.956325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:34.956302    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f38a2d5-c514-4d0d-b714-5149065e6e4b-kube-api-access-9g6vm" pod "1f38a2d5-c514-4d0d-b714-5149065e6e4b" (UID: "1f38a2d5-c514-4d0d-b714-5149065e6e4b"). InnerVolumeSpecName "kube-api-access-9g6vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:34.956325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:34.956303    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-xks-ca" pod "1f38a2d5-c514-4d0d-b714-5149065e6e4b" (UID: "1f38a2d5-c514-4d0d-b714-5149065e6e4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:35.055686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:35.055649    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f38a2d5-c514-4d0d-b714-5149065e6e4b-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:35.055686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:35.055681    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6vm\" (UniqueName: \"kubernetes.io/projected/1f38a2d5-c514-4d0d-b714-5149065e6e4b-kube-api-access-9g6vm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:35.694044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:35.694009    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"
Oct 30 21:19:35.694044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:35.694031    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8" event={"ID":"1f38a2d5-c514-4d0d-b714-5149065e6e4b","Type":"ContainerDied","Data":"b5ffb037eeb45c91bf2398dd49ec32fc8235566c14dc877946c83e7d3bda41c4"}
Oct 30 21:19:35.694275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:35.694092    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5ffb037eeb45c91bf2398dd49ec32fc8235566c14dc877946c83e7d3bda41c4"
Oct 30 21:19:36.003204 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.003107    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj" podUID="0278d2bb-4e87-47aa-8daa-935f1bc41ed0" containerName="payload-processing" containerID="cri-o://372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9" gracePeriod=25
Oct 30 21:19:36.306336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.306304    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:19:36.377256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.377216    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-routing-overlay-volume\") pod \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") "
Oct 30 21:19:36.377473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.377313    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-plugins-config-volume\") pod \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") "
Oct 30 21:19:36.377473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.377416    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-kube-api-access-jqh6p\" (UniqueName: \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-kube-api-access-jqh6p\") pod \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") "
Oct 30 21:19:36.377637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.377618    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-provider-credentials\") pod \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\" (UID: \"0278d2bb-4e87-47aa-8daa-935f1bc41ed0\") "
Oct 30 21:19:36.377700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.377654    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-routing-overlay-volume" pod "0278d2bb-4e87-47aa-8daa-935f1bc41ed0" (UID: "0278d2bb-4e87-47aa-8daa-935f1bc41ed0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:36.379019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.378109    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-plugins-config-volume" pod "0278d2bb-4e87-47aa-8daa-935f1bc41ed0" (UID: "0278d2bb-4e87-47aa-8daa-935f1bc41ed0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:36.379019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.378260    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:36.379019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.378280    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:36.380844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.380817    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-kube-api-access-jqh6p" pod "0278d2bb-4e87-47aa-8daa-935f1bc41ed0" (UID: "0278d2bb-4e87-47aa-8daa-935f1bc41ed0"). InnerVolumeSpecName "kube-api-access-jqh6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:36.380999 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.380898    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-provider-credentials" pod "0278d2bb-4e87-47aa-8daa-935f1bc41ed0" (UID: "0278d2bb-4e87-47aa-8daa-935f1bc41ed0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:36.479689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.479649    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqh6p\" (UniqueName: \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-kube-api-access-jqh6p\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:36.479689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.479683    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0278d2bb-4e87-47aa-8daa-935f1bc41ed0-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:36.705790 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.705751    4810 generic.go:356] "Generic (PLEG): container finished" podID="0278d2bb-4e87-47aa-8daa-935f1bc41ed0" containerID="372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9" exitCode=0
Oct 30 21:19:36.706015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.705809    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"
Oct 30 21:19:36.706015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.705827    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj" event={"ID":"0278d2bb-4e87-47aa-8daa-935f1bc41ed0","Type":"ContainerDied","Data":"372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9"}
Oct 30 21:19:36.706015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.705924    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj" event={"ID":"0278d2bb-4e87-47aa-8daa-935f1bc41ed0","Type":"ContainerDied","Data":"608d886a60a0eeaf333e137bb070a4aa720d0aacfa63d8db1daaffd817ce2766"}
Oct 30 21:19:36.706015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.705964    4810 scope.go:122] "RemoveContainer" containerID="372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9"
Oct 30 21:19:36.719806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.719779    4810 scope.go:122] "RemoveContainer" containerID="372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9"
Oct 30 21:19:36.720129 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:19:36.720107    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9\": container with ID starting with 372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9 not found: ID does not exist" containerID="372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9"
Oct 30 21:19:36.720192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.720139    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9"} err="failed to get container status \"372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9\": rpc error: code = NotFound desc = could not find container \"372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9\": container with ID starting with 372f28673085a52894a0886438abb631addf8fcb9bc2b305c9f402126e881fc9 not found: ID does not exist"
Oct 30 21:19:36.732403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.732372    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"]
Oct 30 21:19:36.739198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:36.739174    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-698bcmxtj"]
Oct 30 21:19:37.663899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:37.663850    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0278d2bb-4e87-47aa-8daa-935f1bc41ed0" path="/var/lib/kubelet/pods/0278d2bb-4e87-47aa-8daa-935f1bc41ed0/volumes"
Oct 30 21:19:42.622838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:42.622798    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"]
Oct 30 21:19:47.630210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:47.626900    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg" podUID="efeecd51-da58-4b3d-b956-569ffd6e868f" containerName="payload-processing" containerID="cri-o://2fe22f96e236bd2d835d061fa2db35f38ef1e4e4565239cdff767f68fd469a2c" gracePeriod=25
Oct 30 21:19:47.841891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:47.841413    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg" event={"ID":"efeecd51-da58-4b3d-b956-569ffd6e868f","Type":"ContainerDied","Data":"2fe22f96e236bd2d835d061fa2db35f38ef1e4e4565239cdff767f68fd469a2c"}
Oct 30 21:19:47.841891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:47.841387    4810 generic.go:356] "Generic (PLEG): container finished" podID="efeecd51-da58-4b3d-b956-569ffd6e868f" containerID="2fe22f96e236bd2d835d061fa2db35f38ef1e4e4565239cdff767f68fd469a2c" exitCode=0
Oct 30 21:19:47.924933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:47.924903    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:19:48.080978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.080946    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-plugins-config-volume\") pod \"efeecd51-da58-4b3d-b956-569ffd6e868f\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") "
Oct 30 21:19:48.081183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.081034    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-routing-overlay-volume\") pod \"efeecd51-da58-4b3d-b956-569ffd6e868f\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") "
Oct 30 21:19:48.081183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.081122    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-kube-api-access-7r4kq\" (UniqueName: \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-kube-api-access-7r4kq\") pod \"efeecd51-da58-4b3d-b956-569ffd6e868f\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") "
Oct 30 21:19:48.081183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.081167    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-provider-credentials\") pod \"efeecd51-da58-4b3d-b956-569ffd6e868f\" (UID: \"efeecd51-da58-4b3d-b956-569ffd6e868f\") "
Oct 30 21:19:48.081464 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.081439    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-routing-overlay-volume" pod "efeecd51-da58-4b3d-b956-569ffd6e868f" (UID: "efeecd51-da58-4b3d-b956-569ffd6e868f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:48.081541 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.081437    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-plugins-config-volume" pod "efeecd51-da58-4b3d-b956-569ffd6e868f" (UID: "efeecd51-da58-4b3d-b956-569ffd6e868f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:48.081840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.081814    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:48.082038 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.082014    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efeecd51-da58-4b3d-b956-569ffd6e868f-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:48.083430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.083400    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-provider-credentials" pod "efeecd51-da58-4b3d-b956-569ffd6e868f" (UID: "efeecd51-da58-4b3d-b956-569ffd6e868f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:48.083530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.083469    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-kube-api-access-7r4kq" pod "efeecd51-da58-4b3d-b956-569ffd6e868f" (UID: "efeecd51-da58-4b3d-b956-569ffd6e868f"). InnerVolumeSpecName "kube-api-access-7r4kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:48.184047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.183954    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:48.184047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.183991    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r4kq\" (UniqueName: \"kubernetes.io/projected/efeecd51-da58-4b3d-b956-569ffd6e868f-kube-api-access-7r4kq\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:48.851114 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.851078    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"
Oct 30 21:19:48.851114 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.851094    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg" event={"ID":"efeecd51-da58-4b3d-b956-569ffd6e868f","Type":"ContainerDied","Data":"3a3e6144f58d04ad63f11d0f3b2a18bc797baa1f22c3e5bce62ea161ef8f1a90"}
Oct 30 21:19:48.851610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.851160    4810 scope.go:122] "RemoveContainer" containerID="2fe22f96e236bd2d835d061fa2db35f38ef1e4e4565239cdff767f68fd469a2c"
Oct 30 21:19:48.880733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.880696    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"]
Oct 30 21:19:48.885304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:48.885277    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-76d8g52bg"]
Oct 30 21:19:49.663961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:49.663931    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efeecd51-da58-4b3d-b956-569ffd6e868f" path="/var/lib/kubelet/pods/efeecd51-da58-4b3d-b956-569ffd6e868f/volumes"
Oct 30 21:19:52.359630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.359592    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"]
Oct 30 21:19:52.360421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.360401    4810 cpu_manager.go:414] "containerMap: removing container" podUID="0278d2bb-4e87-47aa-8daa-935f1bc41ed0" containerName="payload-processing"
Oct 30 21:19:52.360421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.360418    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0278d2bb-4e87-47aa-8daa-935f1bc41ed0" containerName="payload-processing"
Oct 30 21:19:52.360565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.360447    4810 cpu_manager.go:414] "containerMap: removing container" podUID="efeecd51-da58-4b3d-b956-569ffd6e868f" containerName="payload-processing"
Oct 30 21:19:52.360565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.360453    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efeecd51-da58-4b3d-b956-569ffd6e868f" containerName="payload-processing"
Oct 30 21:19:52.360565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.360462    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1f38a2d5-c514-4d0d-b714-5149065e6e4b" containerName="revoke-keys"
Oct 30 21:19:52.360565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.360468    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f38a2d5-c514-4d0d-b714-5149065e6e4b" containerName="revoke-keys"
Oct 30 21:19:52.360693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.360581    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="efeecd51-da58-4b3d-b956-569ffd6e868f" containerName="payload-processing"
Oct 30 21:19:52.360693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.360597    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f38a2d5-c514-4d0d-b714-5149065e6e4b" containerName="revoke-keys"
Oct 30 21:19:52.360693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.360618    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="0278d2bb-4e87-47aa-8daa-935f1bc41ed0" containerName="payload-processing"
Oct 30 21:19:52.365976 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.365958    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.377595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.377557    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmkvd\" (UniqueName: \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-kube-api-access-tmkvd\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.377723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.377612    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.377805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.377733    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.377971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.377845    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.381398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.381104    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"agc-cred-aaf84e248242\""
Oct 30 21:19:52.381398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.381153    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"routing-overlay\""
Oct 30 21:19:52.381398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.381193    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"kube-root-ca.crt\""
Oct 30 21:19:52.381398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.381242    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99668e\""
Oct 30 21:19:52.381398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.381253    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"openshift-service-ca.crt\""
Oct 30 21:19:52.383973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.383952    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"]
Oct 30 21:19:52.479306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.479271    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmkvd\" (UniqueName: \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-kube-api-access-tmkvd\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.479483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.479328    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.479483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.479459    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.479612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.479583    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.480345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.480311    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.480468 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.480422    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.481975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.481950    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.489007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.488927    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmkvd\" (UniqueName: \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-kube-api-access-tmkvd\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.690880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.690122    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:52.989766 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:19:52.989736    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0705b7_c400_49b1_93aa_c9019cfec26f.slice/crio-c06619bcf95325f77d7d38ef963851f12e4488ff3a89537cf8bf1dae928a41ab WatchSource:0}: Error finding container c06619bcf95325f77d7d38ef963851f12e4488ff3a89537cf8bf1dae928a41ab: Status 404 returned error can't find the container with id c06619bcf95325f77d7d38ef963851f12e4488ff3a89537cf8bf1dae928a41ab
Oct 30 21:19:52.991121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:52.991090    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"]
Oct 30 21:19:53.021624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.021599    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"]
Oct 30 21:19:53.027624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.027606    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.033229 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.033204    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e07442\""
Oct 30 21:19:53.033672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.033648    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"openshift-service-ca.crt\""
Oct 30 21:19:53.033672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.033668    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"kube-root-ca.crt\""
Oct 30 21:19:53.033804 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.033692    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"agc-cred-1d2e612c02f2\""
Oct 30 21:19:53.033804 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.033739    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"routing-overlay\""
Oct 30 21:19:53.033911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.033884    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"payload-processing-external-model-e2e--4fdc1812-dockercfg-7shxj\""
Oct 30 21:19:53.034086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.034067    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"]
Oct 30 21:19:53.098658 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.098627    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd67q\" (UniqueName: \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-kube-api-access-nd67q\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.098890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.098852    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.099056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.099039    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.099235 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.099218    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.218712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.218670    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd67q\" (UniqueName: \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-kube-api-access-nd67q\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.219418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.219385    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.238946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.228966    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.238946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.229087    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.238946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.220565    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.238946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.229890    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.238946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.238385    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.257878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.254568    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd67q\" (UniqueName: \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-kube-api-access-nd67q\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.344669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.344638    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.650084 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:19:53.650054    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6020baf4_44d0_4e07_ba66_1b25a844fab1.slice/crio-cba1426a7fcf06490e166af1e3ee98dc8d4b8c9a0e2232b678906aa2cc7bf498 WatchSource:0}: Error finding container cba1426a7fcf06490e166af1e3ee98dc8d4b8c9a0e2232b678906aa2cc7bf498: Status 404 returned error can't find the container with id cba1426a7fcf06490e166af1e3ee98dc8d4b8c9a0e2232b678906aa2cc7bf498
Oct 30 21:19:53.650415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.650349    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"]
Oct 30 21:19:53.951514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.951408    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv" event={"ID":"cf0705b7-c400-49b1-93aa-c9019cfec26f","Type":"ContainerStarted","Data":"94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86"}
Oct 30 21:19:53.951514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.951481    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv" event={"ID":"cf0705b7-c400-49b1-93aa-c9019cfec26f","Type":"ContainerStarted","Data":"c06619bcf95325f77d7d38ef963851f12e4488ff3a89537cf8bf1dae928a41ab"}
Oct 30 21:19:53.951783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.951635    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:19:53.953446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.953423    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq" event={"ID":"6020baf4-44d0-4e07-ba66-1b25a844fab1","Type":"ContainerStarted","Data":"f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e"}
Oct 30 21:19:53.953558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.953466    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:19:53.953558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.953482    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq" event={"ID":"6020baf4-44d0-4e07-ba66-1b25a844fab1","Type":"ContainerStarted","Data":"cba1426a7fcf06490e166af1e3ee98dc8d4b8c9a0e2232b678906aa2cc7bf498"}
Oct 30 21:19:53.975056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.975012    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv" podStartSLOduration=1.97499908 podStartE2EDuration="1.97499908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:53.969053088 +0000 UTC m=+2560.460869633" watchObservedRunningTime="2026-10-30 21:19:53.97499908 +0000 UTC m=+2560.466815626"
Oct 30 21:19:53.986722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:53.986669    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq" podStartSLOduration=1.986652315 podStartE2EDuration="1.986652315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:53.984932358 +0000 UTC m=+2560.476748903" watchObservedRunningTime="2026-10-30 21:19:53.986652315 +0000 UTC m=+2560.478468860"
Oct 30 21:19:55.157800 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:19:55.157711    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:19:59.344007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.343960    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"]
Oct 30 21:19:59.350732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.350704    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"]
Oct 30 21:19:59.351008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.350847    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.359108 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.359082    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"agc-cred-c4141d0965ae\""
Oct 30 21:19:59.359281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.359115    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"payload-processing-external-model-e2e--789fdc40-dockercfg-5ll48\""
Oct 30 21:19:59.464385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.464347    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.464556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.464402    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.464556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.464480    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m22j\" (UniqueName: \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-kube-api-access-7m22j\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.464556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.464527    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.566392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.566353    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.566577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.566403    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.566577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.566441    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m22j\" (UniqueName: \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-kube-api-access-7m22j\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.566577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.566481    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.567344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.567312    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.567497 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.567388    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.569115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.569088    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.575168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.575148    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m22j\" (UniqueName: \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-kube-api-access-7m22j\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.613703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.613640    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"]
Oct 30 21:19:59.618716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.618697    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.622332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.622293    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"agc-cred-34e8e8739776\""
Oct 30 21:19:59.625120 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.625099    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"]
Oct 30 21:19:59.682829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.682793    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:19:59.774033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.771923    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.774033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.772120    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.774033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.772334    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.774033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.772406    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbk52\" (UniqueName: \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-kube-api-access-kbk52\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.873733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.873630    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.873733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.873691    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbk52\" (UniqueName: \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-kube-api-access-kbk52\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.874030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.873742    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.874030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.873842    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.874510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.874488    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.874636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.874616    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.876297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.876276    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.882002 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.881980    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbk52\" (UniqueName: \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-kube-api-access-kbk52\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.942867 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.942828    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:19:59.979384 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:19:59.979356    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa17e4b_b270_4208_bb08_b6ab9910a204.slice/crio-83ce835b4e7afe1e1382f9f7193c944763dfe20ec3842019932c1aac5f76f416 WatchSource:0}: Error finding container 83ce835b4e7afe1e1382f9f7193c944763dfe20ec3842019932c1aac5f76f416: Status 404 returned error can't find the container with id 83ce835b4e7afe1e1382f9f7193c944763dfe20ec3842019932c1aac5f76f416
Oct 30 21:19:59.980600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:19:59.980574    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"]
Oct 30 21:20:00.056899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:00.054200    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5" event={"ID":"efa17e4b-b270-4208-bb08-b6ab9910a204","Type":"ContainerStarted","Data":"83ce835b4e7afe1e1382f9f7193c944763dfe20ec3842019932c1aac5f76f416"}
Oct 30 21:20:00.229477 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:00.229450    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"]
Oct 30 21:20:00.229611 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:20:00.229562    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4bf0bd_8388_4b15_bed9_679f1dab2da7.slice/crio-d1821388ae1add8cd66778fbc4be6ea94f7bb272f14541dcf1182b51a297bc88 WatchSource:0}: Error finding container d1821388ae1add8cd66778fbc4be6ea94f7bb272f14541dcf1182b51a297bc88: Status 404 returned error can't find the container with id d1821388ae1add8cd66778fbc4be6ea94f7bb272f14541dcf1182b51a297bc88
Oct 30 21:20:01.064359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:01.064327    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5" event={"ID":"efa17e4b-b270-4208-bb08-b6ab9910a204","Type":"ContainerStarted","Data":"73b0d0eb6da79f5728f5df264cf051de3826c6c4cd86c9e3403325a3ef13467d"}
Oct 30 21:20:01.064768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:01.064399    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:20:01.066176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:01.066150    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc" event={"ID":"bb4bf0bd-8388-4b15-bed9-679f1dab2da7","Type":"ContainerStarted","Data":"22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89"}
Oct 30 21:20:01.066286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:01.066195    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc" event={"ID":"bb4bf0bd-8388-4b15-bed9-679f1dab2da7","Type":"ContainerStarted","Data":"d1821388ae1add8cd66778fbc4be6ea94f7bb272f14541dcf1182b51a297bc88"}
Oct 30 21:20:01.066286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:01.066236    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:20:01.086306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:01.086264    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5" podStartSLOduration=2.086251375 podStartE2EDuration="2.086251375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:01.081414417 +0000 UTC m=+2567.573230963" watchObservedRunningTime="2026-10-30 21:20:01.086251375 +0000 UTC m=+2567.578067920"
Oct 30 21:20:01.102257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:01.102202    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc" podStartSLOduration=2.102186109 podStartE2EDuration="2.102186109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:01.097761094 +0000 UTC m=+2567.589577639" watchObservedRunningTime="2026-10-30 21:20:01.102186109 +0000 UTC m=+2567.594002655"
Oct 30 21:20:04.965385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:04.965356    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:20:04.965781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:04.965426    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:20:11.625345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:11.625307    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:20:12.077626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:12.077599    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:20:12.077847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:12.077670    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:20:12.140977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:12.140941    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"]
Oct 30 21:20:12.152397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:12.152126    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"]
Oct 30 21:20:17.141404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.141359    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq" podUID="6020baf4-44d0-4e07-ba66-1b25a844fab1" containerName="payload-processing" containerID="cri-o://f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e" gracePeriod=25
Oct 30 21:20:17.153487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.153451    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv" podUID="cf0705b7-c400-49b1-93aa-c9019cfec26f" containerName="payload-processing" containerID="cri-o://94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86" gracePeriod=25
Oct 30 21:20:17.477294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.469726    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:20:17.552881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.538533    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-routing-overlay-volume\") pod \"6020baf4-44d0-4e07-ba66-1b25a844fab1\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") "
Oct 30 21:20:17.552881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.538645    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-kube-api-access-nd67q\" (UniqueName: \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-kube-api-access-nd67q\") pod \"6020baf4-44d0-4e07-ba66-1b25a844fab1\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") "
Oct 30 21:20:17.552881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.538677    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-plugins-config-volume\") pod \"6020baf4-44d0-4e07-ba66-1b25a844fab1\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") "
Oct 30 21:20:17.552881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.538715    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-provider-credentials\") pod \"6020baf4-44d0-4e07-ba66-1b25a844fab1\" (UID: \"6020baf4-44d0-4e07-ba66-1b25a844fab1\") "
Oct 30 21:20:17.552881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.546330    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-routing-overlay-volume" pod "6020baf4-44d0-4e07-ba66-1b25a844fab1" (UID: "6020baf4-44d0-4e07-ba66-1b25a844fab1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:17.552881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.546845    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-plugins-config-volume" pod "6020baf4-44d0-4e07-ba66-1b25a844fab1" (UID: "6020baf4-44d0-4e07-ba66-1b25a844fab1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:17.552881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.548489    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:17.552881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.548522    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6020baf4-44d0-4e07-ba66-1b25a844fab1-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:17.552881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.551573    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-provider-credentials" pod "6020baf4-44d0-4e07-ba66-1b25a844fab1" (UID: "6020baf4-44d0-4e07-ba66-1b25a844fab1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:17.567561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.567520    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-kube-api-access-nd67q" pod "6020baf4-44d0-4e07-ba66-1b25a844fab1" (UID: "6020baf4-44d0-4e07-ba66-1b25a844fab1"). InnerVolumeSpecName "kube-api-access-nd67q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:17.585820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.585792    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:20:17.649219 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.649187    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-plugins-config-volume\") pod \"cf0705b7-c400-49b1-93aa-c9019cfec26f\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") "
Oct 30 21:20:17.649395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.649279    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-provider-credentials\") pod \"cf0705b7-c400-49b1-93aa-c9019cfec26f\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") "
Oct 30 21:20:17.649395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.649333    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-routing-overlay-volume\") pod \"cf0705b7-c400-49b1-93aa-c9019cfec26f\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") "
Oct 30 21:20:17.649531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.649435    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-kube-api-access-tmkvd\" (UniqueName: \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-kube-api-access-tmkvd\") pod \"cf0705b7-c400-49b1-93aa-c9019cfec26f\" (UID: \"cf0705b7-c400-49b1-93aa-c9019cfec26f\") "
Oct 30 21:20:17.649739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.649716    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-plugins-config-volume" pod "cf0705b7-c400-49b1-93aa-c9019cfec26f" (UID: "cf0705b7-c400-49b1-93aa-c9019cfec26f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:17.650046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.650028    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:17.650046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.650036    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-routing-overlay-volume" pod "cf0705b7-c400-49b1-93aa-c9019cfec26f" (UID: "cf0705b7-c400-49b1-93aa-c9019cfec26f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:17.650209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.650047    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd67q\" (UniqueName: \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-kube-api-access-nd67q\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:17.650209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.650086    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6020baf4-44d0-4e07-ba66-1b25a844fab1-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:17.652116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.652086    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-provider-credentials" pod "cf0705b7-c400-49b1-93aa-c9019cfec26f" (UID: "cf0705b7-c400-49b1-93aa-c9019cfec26f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:17.652212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.652138    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-kube-api-access-tmkvd" pod "cf0705b7-c400-49b1-93aa-c9019cfec26f" (UID: "cf0705b7-c400-49b1-93aa-c9019cfec26f"). InnerVolumeSpecName "kube-api-access-tmkvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:17.751386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.751290    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:17.751386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.751328    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf0705b7-c400-49b1-93aa-c9019cfec26f-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:17.751386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.751342    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmkvd\" (UniqueName: \"kubernetes.io/projected/cf0705b7-c400-49b1-93aa-c9019cfec26f-kube-api-access-tmkvd\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:17.807431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.807397    4810 scope.go:122] "RemoveContainer" containerID="4d8462f23ac304bd191f2b8f1f95ca28253bed9c0587041f6ca6fa03132af329"
Oct 30 21:20:17.819303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.819273    4810 scope.go:122] "RemoveContainer" containerID="19a06a6e1d7bfa49d701d07a3af5c350c583c3ceea270aaf3b215184a518c5e8"
Oct 30 21:20:17.830800 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.830774    4810 scope.go:122] "RemoveContainer" containerID="380089cd5b50a47bf24e933478c403443ffe4a30c072cec7ca1920e3facd5cd8"
Oct 30 21:20:17.841423 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:17.841397    4810 scope.go:122] "RemoveContainer" containerID="454dbcadc051761f69f9bade60dd50052abfcd8c6cdaeaa641ecc7ca9feb2ce6"
Oct 30 21:20:18.243966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.243929    4810 generic.go:356] "Generic (PLEG): container finished" podID="6020baf4-44d0-4e07-ba66-1b25a844fab1" containerID="f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e" exitCode=0
Oct 30 21:20:18.244501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.243978    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"
Oct 30 21:20:18.244501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.243996    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq" event={"ID":"6020baf4-44d0-4e07-ba66-1b25a844fab1","Type":"ContainerDied","Data":"f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e"}
Oct 30 21:20:18.244501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.244070    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq" event={"ID":"6020baf4-44d0-4e07-ba66-1b25a844fab1","Type":"ContainerDied","Data":"cba1426a7fcf06490e166af1e3ee98dc8d4b8c9a0e2232b678906aa2cc7bf498"}
Oct 30 21:20:18.244501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.244102    4810 scope.go:122] "RemoveContainer" containerID="f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e"
Oct 30 21:20:18.246781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.246753    4810 generic.go:356] "Generic (PLEG): container finished" podID="cf0705b7-c400-49b1-93aa-c9019cfec26f" containerID="94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86" exitCode=0
Oct 30 21:20:18.246936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.246806    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"
Oct 30 21:20:18.246936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.246834    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv" event={"ID":"cf0705b7-c400-49b1-93aa-c9019cfec26f","Type":"ContainerDied","Data":"94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86"}
Oct 30 21:20:18.247060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.246973    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv" event={"ID":"cf0705b7-c400-49b1-93aa-c9019cfec26f","Type":"ContainerDied","Data":"c06619bcf95325f77d7d38ef963851f12e4488ff3a89537cf8bf1dae928a41ab"}
Oct 30 21:20:18.269963 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.269919    4810 scope.go:122] "RemoveContainer" containerID="f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e"
Oct 30 21:20:18.270606 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:20:18.270573    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e\": container with ID starting with f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e not found: ID does not exist" containerID="f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e"
Oct 30 21:20:18.270747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.270618    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e"} err="failed to get container status \"f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e\": rpc error: code = NotFound desc = could not find container \"f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e\": container with ID starting with f6e8f448a69f1f117b228bb23f9ed8de14be19c19b8effdebadae98378c25b4e not found: ID does not exist"
Oct 30 21:20:18.270747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.270643    4810 scope.go:122] "RemoveContainer" containerID="94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86"
Oct 30 21:20:18.271628 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.271604    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"]
Oct 30 21:20:18.281374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.281350    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776z9wxq"]
Oct 30 21:20:18.284837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.284818    4810 scope.go:122] "RemoveContainer" containerID="94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86"
Oct 30 21:20:18.285257 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:20:18.285235    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86\": container with ID starting with 94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86 not found: ID does not exist" containerID="94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86"
Oct 30 21:20:18.285358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.285264    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86"} err="failed to get container status \"94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86\": rpc error: code = NotFound desc = could not find container \"94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86\": container with ID starting with 94d483f883f12dbbda208c49c1144e3df1876074dddba4e8908583f6000a4d86 not found: ID does not exist"
Oct 30 21:20:18.287785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.287764    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"]
Oct 30 21:20:18.296488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:18.296455    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fm2npv"]
Oct 30 21:20:19.663905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:19.663872    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6020baf4-44d0-4e07-ba66-1b25a844fab1" path="/var/lib/kubelet/pods/6020baf4-44d0-4e07-ba66-1b25a844fab1/volumes"
Oct 30 21:20:19.664350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:19.664336    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0705b7-c400-49b1-93aa-c9019cfec26f" path="/var/lib/kubelet/pods/cf0705b7-c400-49b1-93aa-c9019cfec26f/volumes"
Oct 30 21:20:25.158818 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:20:25.158784    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:20:35.027945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:35.027902    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"]
Oct 30 21:20:35.032420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:35.032390    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cbe369a-h4npm"]
Oct 30 21:20:35.680446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:35.679782    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eafb34c-4a43-446d-89c8-abe3e77334c2" path="/var/lib/kubelet/pods/3eafb34c-4a43-446d-89c8-abe3e77334c2/volumes"
Oct 30 21:20:46.026479 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:46.026442    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"]
Oct 30 21:20:46.030847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:46.030815    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70718462-229nw"]
Oct 30 21:20:47.664494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:47.664461    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ad7868-2e30-4e86-abe5-b6cb67b4f5ab" path="/var/lib/kubelet/pods/09ad7868-2e30-4e86-abe5-b6cb67b4f5ab/volumes"
Oct 30 21:20:52.025987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:52.025950    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"]
Oct 30 21:20:52.029939 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:52.029913    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf773bc2-zt7d5"]
Oct 30 21:20:53.664753 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:20:53.664717    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1b4580-be1b-4e7c-b482-bbc89d2abb30" path="/var/lib/kubelet/pods/fe1b4580-be1b-4e7c-b482-bbc89d2abb30/volumes"
Oct 30 21:20:55.158560 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:20:55.158518    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:21:03.025996 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:03.025955    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"]
Oct 30 21:21:03.030082 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:03.030052    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18eed34f-zsvsl"]
Oct 30 21:21:03.663750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:03.663708    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3b2b0c-5537-4097-bb14-b3fd13729463" path="/var/lib/kubelet/pods/dd3b2b0c-5537-4097-bb14-b3fd13729463/volumes"
Oct 30 21:21:07.026310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:07.026269    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"]
Oct 30 21:21:07.030395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:07.030365    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c218e591-h8pk7"]
Oct 30 21:21:07.664824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:07.664790    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61031fdc-1d64-42e2-bb34-69234ff4b0fe" path="/var/lib/kubelet/pods/61031fdc-1d64-42e2-bb34-69234ff4b0fe/volumes"
Oct 30 21:21:08.025659 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:08.025573    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"]
Oct 30 21:21:08.031013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:08.030981    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-781a71ad-67rxk"]
Oct 30 21:21:09.027341 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:09.027306    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"]
Oct 30 21:21:09.031459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:09.031416    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50db59f0-6q7zt"]
Oct 30 21:21:09.664066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:09.664031    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441766c7-d7c8-4112-99d1-67ff7d064957" path="/var/lib/kubelet/pods/441766c7-d7c8-4112-99d1-67ff7d064957/volumes"
Oct 30 21:21:09.664524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:09.664510    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a13c6af-bfa7-4c2d-a78f-990ca7f451d7" path="/var/lib/kubelet/pods/7a13c6af-bfa7-4c2d-a78f-990ca7f451d7/volumes"
Oct 30 21:21:15.031853 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:15.031819    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"]
Oct 30 21:21:15.037226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:15.037198    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5461e71-mwj28"]
Oct 30 21:21:15.665707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:15.665670    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efb4d17-754a-4d67-addd-cefbe39d96ed" path="/var/lib/kubelet/pods/4efb4d17-754a-4d67-addd-cefbe39d96ed/volumes"
Oct 30 21:21:17.910151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:17.910089    4810 scope.go:122] "RemoveContainer" containerID="46bb17c2354a4e18dfef3881690916874621fa9ba2ae44920065c903d3bd1ea9"
Oct 30 21:21:18.031478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:18.031440    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"]
Oct 30 21:21:18.035336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:18.035301    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1df15ae1-bbln6"]
Oct 30 21:21:19.258669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:19.258634    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:21:19.664116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:19.664073    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb6d5c43-5f90-4877-bdc5-5772a11c9d6d" path="/var/lib/kubelet/pods/eb6d5c43-5f90-4877-bdc5-5772a11c9d6d/volumes"
Oct 30 21:21:23.731786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.731742    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"]
Oct 30 21:21:23.733123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.733097    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6020baf4-44d0-4e07-ba66-1b25a844fab1" containerName="payload-processing"
Oct 30 21:21:23.733233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.733124    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6020baf4-44d0-4e07-ba66-1b25a844fab1" containerName="payload-processing"
Oct 30 21:21:23.733233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.733141    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cf0705b7-c400-49b1-93aa-c9019cfec26f" containerName="payload-processing"
Oct 30 21:21:23.733233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.733149    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0705b7-c400-49b1-93aa-c9019cfec26f" containerName="payload-processing"
Oct 30 21:21:23.733403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.733388    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6020baf4-44d0-4e07-ba66-1b25a844fab1" containerName="payload-processing"
Oct 30 21:21:23.733443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.733410    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0705b7-c400-49b1-93aa-c9019cfec26f" containerName="payload-processing"
Oct 30 21:21:23.740004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.739980    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.746481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.746454    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"]
Oct 30 21:21:23.747771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.747252    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"agc-cred-2c89ff3893b3\""
Oct 30 21:21:23.861329 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.861281    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.861329 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.861326    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.861706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.861679    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn82x\" (UniqueName: \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-kube-api-access-hn82x\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.861791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.861738    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.962372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.962338    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn82x\" (UniqueName: \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-kube-api-access-hn82x\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.962564 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.962387    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.962564 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.962436    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.962692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.962591    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.963328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.963300    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.963472 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.963337    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.965338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.965313    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:23.974608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:23.974582    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn82x\" (UniqueName: \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-kube-api-access-hn82x\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:24.041172 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.041102    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"]
Oct 30 21:21:24.046960 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.046935    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.054308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.053814    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"agc-cred-2eef0b33769b\""
Oct 30 21:21:24.054308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.053951    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"]
Oct 30 21:21:24.128563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.128529    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:24.167762 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.167712    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9d2n\" (UniqueName: \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-kube-api-access-p9d2n\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.167937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.167887    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.168010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.167960    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.168010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.167995    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.269938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.269852    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9d2n\" (UniqueName: \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-kube-api-access-p9d2n\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.270133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.269965    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.270133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.270010    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.270133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.270038    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.271001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.270970    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.275891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.272167    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.275891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.274412    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.279740 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.279710    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9d2n\" (UniqueName: \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-kube-api-access-p9d2n\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.375774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.375691    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:24.434788 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:21:24.434210    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be8a820_9784_4bde_acf6_9a21ea4d0865.slice/crio-fcb08092a18cb2ccc837c0da9d4ff80be2b3864c95e95b3a6a30b5e169b5762b WatchSource:0}: Error finding container fcb08092a18cb2ccc837c0da9d4ff80be2b3864c95e95b3a6a30b5e169b5762b: Status 404 returned error can't find the container with id fcb08092a18cb2ccc837c0da9d4ff80be2b3864c95e95b3a6a30b5e169b5762b
Oct 30 21:21:24.435027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.434921    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"]
Oct 30 21:21:24.675290 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:24.675255    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"]
Oct 30 21:21:24.677998 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:21:24.677972    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdb899c_0580_4942_9573_24134c4cc035.slice/crio-d0f982f8fa94f39330f1cf9c055e5bd0c6831ed4391e4c723c09aec4bb9429c0 WatchSource:0}: Error finding container d0f982f8fa94f39330f1cf9c055e5bd0c6831ed4391e4c723c09aec4bb9429c0: Status 404 returned error can't find the container with id d0f982f8fa94f39330f1cf9c055e5bd0c6831ed4391e4c723c09aec4bb9429c0
Oct 30 21:21:25.029148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:25.029044    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8" event={"ID":"5cdb899c-0580-4942-9573-24134c4cc035","Type":"ContainerStarted","Data":"d60e95d5f1f5fd6cc5744f7a6d2aed0ae442e516e97a87c9ae74dc70217599ea"}
Oct 30 21:21:25.029722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:25.029156    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8" event={"ID":"5cdb899c-0580-4942-9573-24134c4cc035","Type":"ContainerStarted","Data":"d0f982f8fa94f39330f1cf9c055e5bd0c6831ed4391e4c723c09aec4bb9429c0"}
Oct 30 21:21:25.029722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:25.029192    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:25.032314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:25.032284    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b" event={"ID":"5be8a820-9784-4bde-acf6-9a21ea4d0865","Type":"ContainerStarted","Data":"834e7855557a13acad195819312bb492c99230f76665e43b6dc1dea1239da02e"}
Oct 30 21:21:25.032442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:25.032336    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b" event={"ID":"5be8a820-9784-4bde-acf6-9a21ea4d0865","Type":"ContainerStarted","Data":"fcb08092a18cb2ccc837c0da9d4ff80be2b3864c95e95b3a6a30b5e169b5762b"}
Oct 30 21:21:25.032442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:25.032387    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:25.057349 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:25.057277    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8" podStartSLOduration=1.057256539 podStartE2EDuration="1.057256539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:25.055162425 +0000 UTC m=+2651.546978969" watchObservedRunningTime="2026-10-30 21:21:25.057256539 +0000 UTC m=+2651.549073084"
Oct 30 21:21:25.073906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:25.073834    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b" podStartSLOduration=2.073819235 podStartE2EDuration="2.073819235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:25.071727255 +0000 UTC m=+2651.563543783" watchObservedRunningTime="2026-10-30 21:21:25.073819235 +0000 UTC m=+2651.565635774"
Oct 30 21:21:25.158613 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:21:25.158578    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:21:27.025127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:27.025091    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"]
Oct 30 21:21:27.029041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:27.029016    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e3a0111-4l2mp"]
Oct 30 21:21:27.664372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:27.664340    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d2e826-c5fc-4832-8fff-31f6cd1e0561" path="/var/lib/kubelet/pods/61d2e826-c5fc-4832-8fff-31f6cd1e0561/volumes"
Oct 30 21:21:31.028090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:31.028044    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"]
Oct 30 21:21:31.034124 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:31.034093    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91a97f8-68pnd"]
Oct 30 21:21:31.664089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:31.664055    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17" path="/var/lib/kubelet/pods/e860e0db-5f7e-4cb0-8a17-d1fa7d3edf17/volumes"
Oct 30 21:21:36.044235 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:36.044202    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:21:36.044679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:36.044285    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:21:36.118168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:36.118123    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"]
Oct 30 21:21:36.125728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:36.125686    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"]
Oct 30 21:21:41.118439 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.118401    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd" podUID="cadf8d51-ac66-44d4-9428-e17018c70940" containerName="payload-processing" containerID="cri-o://a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe" gracePeriod=25
Oct 30 21:21:41.126939 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.126908    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx" podUID="1797ffd5-a4e6-4064-a217-76c29d658340" containerName="payload-processing" containerID="cri-o://ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58" gracePeriod=25
Oct 30 21:21:41.268734 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:21:41.268701    4810 logging.go:55] [core] [Channel #7345 SubChannel #7347]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.70:50052", ServerName: "10.132.1.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.70:50052: connect: connection refused"
Oct 30 21:21:41.268734 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:21:41.268701    4810 logging.go:55] [core] [Channel #7346 SubChannel #7348]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.69:50052", ServerName: "10.132.1.69:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.69:50052: connect: connection refused"
Oct 30 21:21:41.531438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.531407    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:21:41.535706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.535679    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:21:41.625299 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.625265    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-routing-overlay-volume\") pod \"cadf8d51-ac66-44d4-9428-e17018c70940\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") "
Oct 30 21:21:41.625506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.625416    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-provider-credentials\") pod \"1797ffd5-a4e6-4064-a217-76c29d658340\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") "
Oct 30 21:21:41.625506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.625472    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-plugins-config-volume\") pod \"1797ffd5-a4e6-4064-a217-76c29d658340\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") "
Oct 30 21:21:41.625653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.625532    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-routing-overlay-volume\") pod \"1797ffd5-a4e6-4064-a217-76c29d658340\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") "
Oct 30 21:21:41.625653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.625612    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-kube-api-access-cgfk8\" (UniqueName: \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-kube-api-access-cgfk8\") pod \"1797ffd5-a4e6-4064-a217-76c29d658340\" (UID: \"1797ffd5-a4e6-4064-a217-76c29d658340\") "
Oct 30 21:21:41.625780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.625653    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-routing-overlay-volume" pod "cadf8d51-ac66-44d4-9428-e17018c70940" (UID: "cadf8d51-ac66-44d4-9428-e17018c70940"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:41.625780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.625666    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-provider-credentials\") pod \"cadf8d51-ac66-44d4-9428-e17018c70940\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") "
Oct 30 21:21:41.625780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.625774    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-plugins-config-volume\") pod \"cadf8d51-ac66-44d4-9428-e17018c70940\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") "
Oct 30 21:21:41.626013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.625992    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-kube-api-access-mkdw4\" (UniqueName: \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-kube-api-access-mkdw4\") pod \"cadf8d51-ac66-44d4-9428-e17018c70940\" (UID: \"cadf8d51-ac66-44d4-9428-e17018c70940\") "
Oct 30 21:21:41.626082 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.626047    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-routing-overlay-volume" pod "1797ffd5-a4e6-4064-a217-76c29d658340" (UID: "1797ffd5-a4e6-4064-a217-76c29d658340"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:41.626647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.626583    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:41.626647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.626602    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:41.626647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.626608    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-plugins-config-volume" pod "cadf8d51-ac66-44d4-9428-e17018c70940" (UID: "cadf8d51-ac66-44d4-9428-e17018c70940"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:41.626915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.626768    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-plugins-config-volume" pod "1797ffd5-a4e6-4064-a217-76c29d658340" (UID: "1797ffd5-a4e6-4064-a217-76c29d658340"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:41.628845 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.628818    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-kube-api-access-cgfk8" pod "1797ffd5-a4e6-4064-a217-76c29d658340" (UID: "1797ffd5-a4e6-4064-a217-76c29d658340"). InnerVolumeSpecName "kube-api-access-cgfk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:41.629370 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.629346    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-kube-api-access-mkdw4" pod "cadf8d51-ac66-44d4-9428-e17018c70940" (UID: "cadf8d51-ac66-44d4-9428-e17018c70940"). InnerVolumeSpecName "kube-api-access-mkdw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:41.629816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.629795    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-provider-credentials" pod "cadf8d51-ac66-44d4-9428-e17018c70940" (UID: "cadf8d51-ac66-44d4-9428-e17018c70940"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:41.629925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.629835    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-provider-credentials" pod "1797ffd5-a4e6-4064-a217-76c29d658340" (UID: "1797ffd5-a4e6-4064-a217-76c29d658340"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:41.728094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.728012    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgfk8\" (UniqueName: \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-kube-api-access-cgfk8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:41.728094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.728038    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:41.728094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.728048    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadf8d51-ac66-44d4-9428-e17018c70940-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:41.728094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.728059    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkdw4\" (UniqueName: \"kubernetes.io/projected/cadf8d51-ac66-44d4-9428-e17018c70940-kube-api-access-mkdw4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:41.728094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.728067    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1797ffd5-a4e6-4064-a217-76c29d658340-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:41.728094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:41.728077    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797ffd5-a4e6-4064-a217-76c29d658340-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:42.204176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.198833    4810 generic.go:356] "Generic (PLEG): container finished" podID="cadf8d51-ac66-44d4-9428-e17018c70940" containerID="a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe" exitCode=0
Oct 30 21:21:42.204176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.199163    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd" event={"ID":"cadf8d51-ac66-44d4-9428-e17018c70940","Type":"ContainerDied","Data":"a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe"}
Oct 30 21:21:42.204176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.199221    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd" event={"ID":"cadf8d51-ac66-44d4-9428-e17018c70940","Type":"ContainerDied","Data":"de5ff3790bdc314462b9b82b561f260a045df1f7607d523eba221f25f0999ef6"}
Oct 30 21:21:42.204176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.199254    4810 scope.go:122] "RemoveContainer" containerID="a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe"
Oct 30 21:21:42.204176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.199421    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"
Oct 30 21:21:42.218149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.209164    4810 generic.go:356] "Generic (PLEG): container finished" podID="1797ffd5-a4e6-4064-a217-76c29d658340" containerID="ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58" exitCode=0
Oct 30 21:21:42.218149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.209241    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx" event={"ID":"1797ffd5-a4e6-4064-a217-76c29d658340","Type":"ContainerDied","Data":"ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58"}
Oct 30 21:21:42.218149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.209306    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx" event={"ID":"1797ffd5-a4e6-4064-a217-76c29d658340","Type":"ContainerDied","Data":"5cd1da2466ec63273a7dc0414d7af8928466b3550caec39d4390664334dd2101"}
Oct 30 21:21:42.218149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.209400    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"
Oct 30 21:21:42.256628 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.256554    4810 scope.go:122] "RemoveContainer" containerID="a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe"
Oct 30 21:21:42.256628 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.256568    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"]
Oct 30 21:21:42.257794 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:21:42.257765    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe\": container with ID starting with a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe not found: ID does not exist" containerID="a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe"
Oct 30 21:21:42.258007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.257978    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe"} err="failed to get container status \"a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe\": rpc error: code = NotFound desc = could not find container \"a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe\": container with ID starting with a5da044d95c983d3adf8d4c3a6e7b4775d91ef488da339fae68dba2ef840dabe not found: ID does not exist"
Oct 30 21:21:42.258119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.258107    4810 scope.go:122] "RemoveContainer" containerID="ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58"
Oct 30 21:21:42.278086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.269473    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx" podUID="1797ffd5-a4e6-4064-a217-76c29d658340" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.69:50052\" within 1s: context deadline exceeded"
Oct 30 21:21:42.278086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.269825    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd" podUID="cadf8d51-ac66-44d4-9428-e17018c70940" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.70:50052\" within 1s: context deadline exceeded"
Oct 30 21:21:42.278086 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:21:42.269972    4810 logging.go:55] [core] [Channel #7346 SubChannel #7348]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.69:50052", ServerName: "10.132.1.69:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.69:50052: operation was canceled"
Oct 30 21:21:42.278086 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:21:42.270032    4810 logging.go:55] [core] [Channel #7345 SubChannel #7347]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.70:50052", ServerName: "10.132.1.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.70:50052: operation was canceled"
Oct 30 21:21:42.282364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.282241    4810 scope.go:122] "RemoveContainer" containerID="ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58"
Oct 30 21:21:42.283514 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:21:42.282878    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58\": container with ID starting with ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58 not found: ID does not exist" containerID="ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58"
Oct 30 21:21:42.283514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.282920    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58"} err="failed to get container status \"ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58\": rpc error: code = NotFound desc = could not find container \"ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58\": container with ID starting with ec1eb83685cc4f92d11c0ddff15c821430c5635f5379a878e6eb40f2f5ea8f58 not found: ID does not exist"
Oct 30 21:21:42.283514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.283433    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697tbmvd"]
Oct 30 21:21:42.290233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.290207    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"]
Oct 30 21:21:42.296518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:42.296496    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjdbxx"]
Oct 30 21:21:43.667011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:43.666980    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1797ffd5-a4e6-4064-a217-76c29d658340" path="/var/lib/kubelet/pods/1797ffd5-a4e6-4064-a217-76c29d658340/volumes"
Oct 30 21:21:43.667407 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:43.667393    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cadf8d51-ac66-44d4-9428-e17018c70940" path="/var/lib/kubelet/pods/cadf8d51-ac66-44d4-9428-e17018c70940/volumes"
Oct 30 21:21:49.939827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.939795    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"]
Oct 30 21:21:49.940584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.940569    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1797ffd5-a4e6-4064-a217-76c29d658340" containerName="payload-processing"
Oct 30 21:21:49.940632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.940585    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1797ffd5-a4e6-4064-a217-76c29d658340" containerName="payload-processing"
Oct 30 21:21:49.940632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.940600    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cadf8d51-ac66-44d4-9428-e17018c70940" containerName="payload-processing"
Oct 30 21:21:49.940632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.940605    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cadf8d51-ac66-44d4-9428-e17018c70940" containerName="payload-processing"
Oct 30 21:21:49.940761 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.940752    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1797ffd5-a4e6-4064-a217-76c29d658340" containerName="payload-processing"
Oct 30 21:21:49.940798 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.940765    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cadf8d51-ac66-44d4-9428-e17018c70940" containerName="payload-processing"
Oct 30 21:21:49.946157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.946138    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:49.950120 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.950095    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:21:49.950508 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.950487    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:21:49.952259 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.952234    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"]
Oct 30 21:21:49.966421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.966391    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d8f8744-t7vqv\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:49.966549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.966445    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtjzl\" (UniqueName: \"kubernetes.io/projected/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-kube-api-access-xtjzl\") pod \"maas-api-revoke-sub-0d8f8744-t7vqv\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:49.966614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:49.966592    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d8f8744-t7vqv\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:50.067836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:50.067802    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d8f8744-t7vqv\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:50.068078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:50.067866    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtjzl\" (UniqueName: \"kubernetes.io/projected/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-kube-api-access-xtjzl\") pod \"maas-api-revoke-sub-0d8f8744-t7vqv\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:50.068078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:50.067936    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d8f8744-t7vqv\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:50.068573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:50.068553    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d8f8744-t7vqv\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:50.070310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:50.070284    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d8f8744-t7vqv\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:50.076640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:50.076612    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtjzl\" (UniqueName: \"kubernetes.io/projected/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-kube-api-access-xtjzl\") pod \"maas-api-revoke-sub-0d8f8744-t7vqv\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:50.271276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:50.271181    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:50.557597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:50.557566    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"]
Oct 30 21:21:50.558671 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:21:50.558643    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda994b6bb_cb12_42e7_b541_e8e4440fbbd8.slice/crio-607772a8af832255c857b1cfc6053bda05fa7e411b0cde905bbe14b0a212163e WatchSource:0}: Error finding container 607772a8af832255c857b1cfc6053bda05fa7e411b0cde905bbe14b0a212163e: Status 404 returned error can't find the container with id 607772a8af832255c857b1cfc6053bda05fa7e411b0cde905bbe14b0a212163e
Oct 30 21:21:51.288451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:51.288362    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv" event={"ID":"a994b6bb-cb12-42e7-b541-e8e4440fbbd8","Type":"ContainerStarted","Data":"a5e6a92a57e50d9c44ea53ca33f46951e784b913be2a8a85e612efe38d377bf0"}
Oct 30 21:21:51.288451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:51.288450    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv" event={"ID":"a994b6bb-cb12-42e7-b541-e8e4440fbbd8","Type":"ContainerStarted","Data":"607772a8af832255c857b1cfc6053bda05fa7e411b0cde905bbe14b0a212163e"}
Oct 30 21:21:51.307945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:51.307890    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv" podStartSLOduration=2.307870025 podStartE2EDuration="2.307870025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:51.303246781 +0000 UTC m=+2677.795063327" watchObservedRunningTime="2026-10-30 21:21:51.307870025 +0000 UTC m=+2677.799686574"
Oct 30 21:21:52.299723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:52.299675    4810 generic.go:356] "Generic (PLEG): container finished" podID="a994b6bb-cb12-42e7-b541-e8e4440fbbd8" containerID="a5e6a92a57e50d9c44ea53ca33f46951e784b913be2a8a85e612efe38d377bf0" exitCode=0
Oct 30 21:21:52.300271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:52.299908    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv" event={"ID":"a994b6bb-cb12-42e7-b541-e8e4440fbbd8","Type":"ContainerDied","Data":"a5e6a92a57e50d9c44ea53ca33f46951e784b913be2a8a85e612efe38d377bf0"}
Oct 30 21:21:53.562034 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:53.561968    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:53.637698 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:53.637662    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-xks-ca\") pod \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") "
Oct 30 21:21:53.637885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:53.637716    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-openshift-ca\") pod \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") "
Oct 30 21:21:53.638122 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:53.638098    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-openshift-ca" pod "a994b6bb-cb12-42e7-b541-e8e4440fbbd8" (UID: "a994b6bb-cb12-42e7-b541-e8e4440fbbd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:53.638265 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:53.638230    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-kube-api-access-xtjzl\" (UniqueName: \"kubernetes.io/projected/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-kube-api-access-xtjzl\") pod \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\" (UID: \"a994b6bb-cb12-42e7-b541-e8e4440fbbd8\") "
Oct 30 21:21:53.639224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:53.639197    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:53.640230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:53.640198    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-xks-ca" pod "a994b6bb-cb12-42e7-b541-e8e4440fbbd8" (UID: "a994b6bb-cb12-42e7-b541-e8e4440fbbd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:21:53.640404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:53.640387    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-kube-api-access-xtjzl" pod "a994b6bb-cb12-42e7-b541-e8e4440fbbd8" (UID: "a994b6bb-cb12-42e7-b541-e8e4440fbbd8"). InnerVolumeSpecName "kube-api-access-xtjzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:53.741250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:53.741218    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtjzl\" (UniqueName: \"kubernetes.io/projected/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-kube-api-access-xtjzl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:53.741390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:53.741254    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a994b6bb-cb12-42e7-b541-e8e4440fbbd8-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:54.319130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:54.319102    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"
Oct 30 21:21:54.319343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:54.319131    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv" event={"ID":"a994b6bb-cb12-42e7-b541-e8e4440fbbd8","Type":"ContainerDied","Data":"607772a8af832255c857b1cfc6053bda05fa7e411b0cde905bbe14b0a212163e"}
Oct 30 21:21:54.319343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:54.319179    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="607772a8af832255c857b1cfc6053bda05fa7e411b0cde905bbe14b0a212163e"
Oct 30 21:21:55.158573 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:21:55.158530    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:21:57.111094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:57.111060    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"]
Oct 30 21:21:57.111528 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:21:57.111289    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" containerID="cri-o://04815a806bc94d7c09edae61cdc2be5886ff138f6c1aece93382909172c65dd3" gracePeriod=30
Oct 30 21:22:01.044885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.043936    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"]
Oct 30 21:22:01.045443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.045281    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a994b6bb-cb12-42e7-b541-e8e4440fbbd8" containerName="revoke-keys"
Oct 30 21:22:01.045443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.045306    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a994b6bb-cb12-42e7-b541-e8e4440fbbd8" containerName="revoke-keys"
Oct 30 21:22:01.045581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.045528    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a994b6bb-cb12-42e7-b541-e8e4440fbbd8" containerName="revoke-keys"
Oct 30 21:22:01.082164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.082126    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"]
Oct 30 21:22:01.082373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.082260    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.174907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.174848    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-provider-credentials\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.175354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.175331    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-plugins-config-volume\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.175512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.175494    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hzsb\" (UniqueName: \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-kube-api-access-4hzsb\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.175791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.175774    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-routing-overlay-volume\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.277386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.277345    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-provider-credentials\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.277549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.277470    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-plugins-config-volume\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.277549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.277501    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hzsb\" (UniqueName: \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-kube-api-access-4hzsb\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.277674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.277655    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-routing-overlay-volume\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.278274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.278247    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-plugins-config-volume\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.278423 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.278387    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-routing-overlay-volume\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.280251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.280227    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-provider-credentials\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.286140 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.286117    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hzsb\" (UniqueName: \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-kube-api-access-4hzsb\") pod \"payload-processing-external-model-9f8bd7d97-kfm4f\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") " pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.429119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.429084    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:01.940189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:01.940159    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"]
Oct 30 21:22:01.941153 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:22:01.941126    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946ceaf4_9142_421e_9532_89c096747923.slice/crio-9b26bd99a4ab9bc4d0cff86748dc939984aa23a87eb6750ba6223e29744019f8 WatchSource:0}: Error finding container 9b26bd99a4ab9bc4d0cff86748dc939984aa23a87eb6750ba6223e29744019f8: Status 404 returned error can't find the container with id 9b26bd99a4ab9bc4d0cff86748dc939984aa23a87eb6750ba6223e29744019f8
Oct 30 21:22:02.103665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.103622    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"]
Oct 30 21:22:02.104220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.103906    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy" containerID="cri-o://fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974" gracePeriod=30
Oct 30 21:22:02.264917 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:02.264769    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5475c236_1e38_45ef_b483_9001b2fbf2f4.slice/crio-04815a806bc94d7c09edae61cdc2be5886ff138f6c1aece93382909172c65dd3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5475c236_1e38_45ef_b483_9001b2fbf2f4.slice/crio-conmon-04815a806bc94d7c09edae61cdc2be5886ff138f6c1aece93382909172c65dd3.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:02.416083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.416052    4810 generic.go:356] "Generic (PLEG): container finished" podID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerID="04815a806bc94d7c09edae61cdc2be5886ff138f6c1aece93382909172c65dd3" exitCode=0
Oct 30 21:22:02.416278 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.416227    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" event={"ID":"5475c236-1e38-45ef-b483-9001b2fbf2f4","Type":"ContainerDied","Data":"04815a806bc94d7c09edae61cdc2be5886ff138f6c1aece93382909172c65dd3"}
Oct 30 21:22:02.418332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.418303    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f" event={"ID":"946ceaf4-9142-421e-9532-89c096747923","Type":"ContainerStarted","Data":"4195fc8eed58ffeb6c9d88740b1f07cfadafc254ba4a1f2ddc5e37fc9cd87b2b"}
Oct 30 21:22:02.418499 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.418355    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f" event={"ID":"946ceaf4-9142-421e-9532-89c096747923","Type":"ContainerStarted","Data":"9b26bd99a4ab9bc4d0cff86748dc939984aa23a87eb6750ba6223e29744019f8"}
Oct 30 21:22:02.418822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.418785    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:02.443616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.443562    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f" podStartSLOduration=2.443547769 podStartE2EDuration="2.443547769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:02.442360716 +0000 UTC m=+2688.934177261" watchObservedRunningTime="2026-10-30 21:22:02.443547769 +0000 UTC m=+2688.935364363"
Oct 30 21:22:02.493442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.493411    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 21:22:02.561983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.561892    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-podinfo\") pod \"5475c236-1e38-45ef-b483-9001b2fbf2f4\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") "
Oct 30 21:22:02.562219 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.562009    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-credential-socket\") pod \"5475c236-1e38-45ef-b483-9001b2fbf2f4\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") "
Oct 30 21:22:02.565068 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.565034    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-credential-socket" pod "5475c236-1e38-45ef-b483-9001b2fbf2f4" (UID: "5475c236-1e38-45ef-b483-9001b2fbf2f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:02.565292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.565264    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-token\" (UniqueName: \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-token\") pod \"5475c236-1e38-45ef-b483-9001b2fbf2f4\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") "
Oct 30 21:22:02.565390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.565314    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-certs\") pod \"5475c236-1e38-45ef-b483-9001b2fbf2f4\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") "
Oct 30 21:22:02.565390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.565339    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-podinfo" pod "5475c236-1e38-45ef-b483-9001b2fbf2f4" (UID: "5475c236-1e38-45ef-b483-9001b2fbf2f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:02.565642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.565621    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-kube-api-access-t7zts\" (UniqueName: \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-kube-api-access-t7zts\") pod \"5475c236-1e38-45ef-b483-9001b2fbf2f4\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") "
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.566143    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-data\") pod \"5475c236-1e38-45ef-b483-9001b2fbf2f4\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") "
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.566206    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-socket\") pod \"5475c236-1e38-45ef-b483-9001b2fbf2f4\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") "
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.567014    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-socket" pod "5475c236-1e38-45ef-b483-9001b2fbf2f4" (UID: "5475c236-1e38-45ef-b483-9001b2fbf2f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.567609    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-data" pod "5475c236-1e38-45ef-b483-9001b2fbf2f4" (UID: "5475c236-1e38-45ef-b483-9001b2fbf2f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.567631    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5475c236-1e38-45ef-b483-9001b2fbf2f4-istiod-ca-cert" pod "5475c236-1e38-45ef-b483-9001b2fbf2f4" (UID: "5475c236-1e38-45ef-b483-9001b2fbf2f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.567648    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5475c236-1e38-45ef-b483-9001b2fbf2f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5475c236-1e38-45ef-b483-9001b2fbf2f4-istiod-ca-cert\") pod \"5475c236-1e38-45ef-b483-9001b2fbf2f4\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") "
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.567708    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-envoy\") pod \"5475c236-1e38-45ef-b483-9001b2fbf2f4\" (UID: \"5475c236-1e38-45ef-b483-9001b2fbf2f4\") "
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.569291    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.569317    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.569331    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5475c236-1e38-45ef-b483-9001b2fbf2f4-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.569345    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.569360    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.569477    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-certs" pod "5475c236-1e38-45ef-b483-9001b2fbf2f4" (UID: "5475c236-1e38-45ef-b483-9001b2fbf2f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.569611    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-token" pod "5475c236-1e38-45ef-b483-9001b2fbf2f4" (UID: "5475c236-1e38-45ef-b483-9001b2fbf2f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.571659    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-kube-api-access-t7zts" pod "5475c236-1e38-45ef-b483-9001b2fbf2f4" (UID: "5475c236-1e38-45ef-b483-9001b2fbf2f4"). InnerVolumeSpecName "kube-api-access-t7zts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:02.589904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.573497    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-envoy" pod "5475c236-1e38-45ef-b483-9001b2fbf2f4" (UID: "5475c236-1e38-45ef-b483-9001b2fbf2f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:02.671181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.671144    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:02.671181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.671176    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:02.671181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.671186    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7zts\" (UniqueName: \"kubernetes.io/projected/5475c236-1e38-45ef-b483-9001b2fbf2f4-kube-api-access-t7zts\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:02.671473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:02.671194    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5475c236-1e38-45ef-b483-9001b2fbf2f4-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:03.408312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:03.408271    4810 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:22:03.408798 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:03.408341    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:22:03.430881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:03.430805    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd" event={"ID":"5475c236-1e38-45ef-b483-9001b2fbf2f4","Type":"ContainerDied","Data":"8d266c44d48d44afe4c25daabe52468f725805aaee373a484878db16156faa43"}
Oct 30 21:22:03.431315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:03.431151    4810 scope.go:122] "RemoveContainer" containerID="04815a806bc94d7c09edae61cdc2be5886ff138f6c1aece93382909172c65dd3"
Oct 30 21:22:03.441978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:03.440940    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"
Oct 30 21:22:03.513404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:03.513374    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"]
Oct 30 21:22:03.520759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:03.520733    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-nvxkd"]
Oct 30 21:22:03.666385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:03.666312    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" path="/var/lib/kubelet/pods/5475c236-1e38-45ef-b483-9001b2fbf2f4/volumes"
Oct 30 21:22:05.163227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:05.163195    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:22:06.990322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:06.990285    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"]
Oct 30 21:22:06.990799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:06.990517    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" podUID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" containerName="istio-proxy" containerID="cri-o://3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e" gracePeriod=30
Oct 30 21:22:06.996550 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:06.996520    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"]
Oct 30 21:22:06.996923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:06.996895    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" podUID="08995227-51dd-4ba8-9bbe-c4bd9877903e" containerName="istio-proxy" containerID="cri-o://01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41" gracePeriod=30
Oct 30 21:22:07.391789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.391764    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 21:22:07.459617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.459583    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95bb7748-246a-49db-9390-9ea9924af937-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bb7748-246a-49db-9390-9ea9924af937-istio-podinfo\") pod \"95bb7748-246a-49db-9390-9ea9924af937\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") "
Oct 30 21:22:07.459617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.459630    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-credential-socket\") pod \"95bb7748-246a-49db-9390-9ea9924af937\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") "
Oct 30 21:22:07.459950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.459657    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95bb7748-246a-49db-9390-9ea9924af937-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bb7748-246a-49db-9390-9ea9924af937-istiod-ca-cert\") pod \"95bb7748-246a-49db-9390-9ea9924af937\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") "
Oct 30 21:22:07.459950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.459821    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-kube-api-access-zz9kk\" (UniqueName: \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-kube-api-access-zz9kk\") pod \"95bb7748-246a-49db-9390-9ea9924af937\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") "
Oct 30 21:22:07.459950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.459896    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-data\") pod \"95bb7748-246a-49db-9390-9ea9924af937\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") "
Oct 30 21:22:07.460144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.460008    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-istio-token\" (UniqueName: \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-istio-token\") pod \"95bb7748-246a-49db-9390-9ea9924af937\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") "
Oct 30 21:22:07.460144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.460045    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-socket\") pod \"95bb7748-246a-49db-9390-9ea9924af937\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") "
Oct 30 21:22:07.460144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.460094    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-envoy\") pod \"95bb7748-246a-49db-9390-9ea9924af937\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") "
Oct 30 21:22:07.460314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.460224    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-certs\") pod \"95bb7748-246a-49db-9390-9ea9924af937\" (UID: \"95bb7748-246a-49db-9390-9ea9924af937\") "
Oct 30 21:22:07.460718 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.460682    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-credential-socket" pod "95bb7748-246a-49db-9390-9ea9924af937" (UID: "95bb7748-246a-49db-9390-9ea9924af937"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:07.460943 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.460916    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:07.461974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.461734    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-socket" pod "95bb7748-246a-49db-9390-9ea9924af937" (UID: "95bb7748-246a-49db-9390-9ea9924af937"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:07.461974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.461907    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95bb7748-246a-49db-9390-9ea9924af937-istiod-ca-cert" pod "95bb7748-246a-49db-9390-9ea9924af937" (UID: "95bb7748-246a-49db-9390-9ea9924af937"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:07.462250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.462200    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-data" pod "95bb7748-246a-49db-9390-9ea9924af937" (UID: "95bb7748-246a-49db-9390-9ea9924af937"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:07.465101 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.465073    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-certs" pod "95bb7748-246a-49db-9390-9ea9924af937" (UID: "95bb7748-246a-49db-9390-9ea9924af937"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:07.466840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.466718    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95bb7748-246a-49db-9390-9ea9924af937-istio-podinfo" pod "95bb7748-246a-49db-9390-9ea9924af937" (UID: "95bb7748-246a-49db-9390-9ea9924af937"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:07.466840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.466736    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-envoy" pod "95bb7748-246a-49db-9390-9ea9924af937" (UID: "95bb7748-246a-49db-9390-9ea9924af937"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:07.466840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.466766    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-istio-token" pod "95bb7748-246a-49db-9390-9ea9924af937" (UID: "95bb7748-246a-49db-9390-9ea9924af937"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:07.466840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.466812    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-kube-api-access-zz9kk" pod "95bb7748-246a-49db-9390-9ea9924af937" (UID: "95bb7748-246a-49db-9390-9ea9924af937"). InnerVolumeSpecName "kube-api-access-zz9kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:07.471706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.471680    4810 generic.go:356] "Generic (PLEG): container finished" podID="95bb7748-246a-49db-9390-9ea9924af937" containerID="fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974" exitCode=0
Oct 30 21:22:07.472049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.472027    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" event={"ID":"95bb7748-246a-49db-9390-9ea9924af937","Type":"ContainerDied","Data":"fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974"}
Oct 30 21:22:07.472142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.472084    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278" event={"ID":"95bb7748-246a-49db-9390-9ea9924af937","Type":"ContainerDied","Data":"03b545900e6d0b0a3d4185e0b5249869cbf823b8c5c80e5f585ff338db9772f9"}
Oct 30 21:22:07.472142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.472112    4810 scope.go:122] "RemoveContainer" containerID="fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974"
Oct 30 21:22:07.472263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.472256    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"
Oct 30 21:22:07.506110 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.506088    4810 scope.go:122] "RemoveContainer" containerID="fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974"
Oct 30 21:22:07.506458 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:07.506428    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974\": container with ID starting with fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974 not found: ID does not exist" containerID="fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974"
Oct 30 21:22:07.506544 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.506470    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974"} err="failed to get container status \"fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974\": rpc error: code = NotFound desc = could not find container \"fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974\": container with ID starting with fec62a52142097b686ea568511bb2b610abad2d0a3977168a98a935907c07974 not found: ID does not exist"
Oct 30 21:22:07.531390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.531300    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"]
Oct 30 21:22:07.544822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.544794    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-mk278"]
Oct 30 21:22:07.561960 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.561934    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95bb7748-246a-49db-9390-9ea9924af937-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:07.561960 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.561959    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95bb7748-246a-49db-9390-9ea9924af937-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:07.562107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.561970    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz9kk\" (UniqueName: \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-kube-api-access-zz9kk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:07.562107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.561980    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:07.562107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.561990    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95bb7748-246a-49db-9390-9ea9924af937-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:07.562107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.561998    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:07.562107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.562006    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:07.562107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.562016    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95bb7748-246a-49db-9390-9ea9924af937-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:07.667316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:07.667288    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95bb7748-246a-49db-9390-9ea9924af937" path="/var/lib/kubelet/pods/95bb7748-246a-49db-9390-9ea9924af937/volumes"
Oct 30 21:22:08.837016 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.836978    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"]
Oct 30 21:22:08.837809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.837789    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy"
Oct 30 21:22:08.837892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.837810    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy"
Oct 30 21:22:08.837892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.837830    4810 cpu_manager.go:414] "containerMap: removing container" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy"
Oct 30 21:22:08.837892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.837836    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy"
Oct 30 21:22:08.838011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.837977    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5475c236-1e38-45ef-b483-9001b2fbf2f4" containerName="istio-proxy"
Oct 30 21:22:08.838011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.837988    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="95bb7748-246a-49db-9390-9ea9924af937" containerName="istio-proxy"
Oct 30 21:22:08.843166 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.843149    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:22:08.850916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.850888    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"]
Oct 30 21:22:08.853488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.853466    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-9a57c4\"/\"openshift-service-ca.crt\""
Oct 30 21:22:08.853629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.853515    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-9a57c4\"/\"kube-root-ca.crt\""
Oct 30 21:22:08.853629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.853555    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-9a57c4\"/\"default-dockercfg-jbbdm\""
Oct 30 21:22:08.853732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.853648    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-9a57c4\"/\"e2e-usage-sim-8fece6-cert\""
Oct 30 21:22:08.979047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.979016    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-tls\") pod \"e2e-usage-sim-8fece6-85bfb989b-2dbld\" (UID: \"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48\") " pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:22:08.979212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:08.979055    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9mx\" (UniqueName: \"kubernetes.io/projected/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-kube-api-access-5r9mx\") pod \"e2e-usage-sim-8fece6-85bfb989b-2dbld\" (UID: \"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48\") " pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:22:09.080228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:09.080198    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-tls\") pod \"e2e-usage-sim-8fece6-85bfb989b-2dbld\" (UID: \"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48\") " pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:22:09.080404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:09.080238    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9mx\" (UniqueName: \"kubernetes.io/projected/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-kube-api-access-5r9mx\") pod \"e2e-usage-sim-8fece6-85bfb989b-2dbld\" (UID: \"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48\") " pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:22:09.082625 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:09.082600    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-tls\") pod \"e2e-usage-sim-8fece6-85bfb989b-2dbld\" (UID: \"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48\") " pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:22:09.089225 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:09.089164    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9mx\" (UniqueName: \"kubernetes.io/projected/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-kube-api-access-5r9mx\") pod \"e2e-usage-sim-8fece6-85bfb989b-2dbld\" (UID: \"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48\") " pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:22:09.158706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:09.158677    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:22:09.488345 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:22:09.488317    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cdebae3_7b99_4d7b_b6bb_c6a30fd53e48.slice/crio-2df11d47c91ae6af65356cfccdfa91bcd7e1d978c880794515d8694c4ceaff3d WatchSource:0}: Error finding container 2df11d47c91ae6af65356cfccdfa91bcd7e1d978c880794515d8694c4ceaff3d: Status 404 returned error can't find the container with id 2df11d47c91ae6af65356cfccdfa91bcd7e1d978c880794515d8694c4ceaff3d
Oct 30 21:22:09.489673 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:09.489646    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"]
Oct 30 21:22:09.542180 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:09.542152    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld" event={"ID":"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48","Type":"ContainerStarted","Data":"2df11d47c91ae6af65356cfccdfa91bcd7e1d978c880794515d8694c4ceaff3d"}
Oct 30 21:22:10.564232 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:10.564167    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld" event={"ID":"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48","Type":"ContainerStarted","Data":"25717b2164ae41385f2c19647057ae19015398b12f8a57ecae5c6ff5fa9b5b09"}
Oct 30 21:22:10.564742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:10.564243    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:22:10.570160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:10.570136    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:22:10.580896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:10.580834    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld" podStartSLOduration=2.580820729 podStartE2EDuration="2.580820729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:10.579114687 +0000 UTC m=+2697.070931235" watchObservedRunningTime="2026-10-30 21:22:10.580820729 +0000 UTC m=+2697.072637274"
Oct 30 21:22:12.421990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.421961    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:22:12.426691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.426667    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:22:12.546279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546242    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-socket\") pod \"08995227-51dd-4ba8-9bbe-c4bd9877903e\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") "
Oct 30 21:22:12.546279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546284    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-kube-api-access-gp97l\" (UniqueName: \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-kube-api-access-gp97l\") pod \"08995227-51dd-4ba8-9bbe-c4bd9877903e\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") "
Oct 30 21:22:12.546547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546313    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-credential-socket\") pod \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") "
Oct 30 21:22:12.546547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546350    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08995227-51dd-4ba8-9bbe-c4bd9877903e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08995227-51dd-4ba8-9bbe-c4bd9877903e-istiod-ca-cert\") pod \"08995227-51dd-4ba8-9bbe-c4bd9877903e\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") "
Oct 30 21:22:12.546547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546387    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-envoy\") pod \"08995227-51dd-4ba8-9bbe-c4bd9877903e\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") "
Oct 30 21:22:12.546547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546435    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-data\") pod \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") "
Oct 30 21:22:12.546547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546462    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-token\" (UniqueName: \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-token\") pod \"08995227-51dd-4ba8-9bbe-c4bd9877903e\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") "
Oct 30 21:22:12.546547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546519    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-socket" pod "08995227-51dd-4ba8-9bbe-c4bd9877903e" (UID: "08995227-51dd-4ba8-9bbe-c4bd9877903e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:12.546547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546539    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-certs\") pod \"08995227-51dd-4ba8-9bbe-c4bd9877903e\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") "
Oct 30 21:22:12.546969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546591    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-podinfo\") pod \"08995227-51dd-4ba8-9bbe-c4bd9877903e\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") "
Oct 30 21:22:12.546969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546646    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-token\" (UniqueName: \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-token\") pod \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") "
Oct 30 21:22:12.546969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546688    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-socket\") pod \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") "
Oct 30 21:22:12.546969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546752    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-envoy\") pod \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") "
Oct 30 21:22:12.546969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546825    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-certs\") pod \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") "
Oct 30 21:22:12.547252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547001    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-podinfo\") pod \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") "
Oct 30 21:22:12.547252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547065    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istiod-ca-cert\") pod \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") "
Oct 30 21:22:12.547252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547110    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-data\") pod \"08995227-51dd-4ba8-9bbe-c4bd9877903e\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") "
Oct 30 21:22:12.547252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547149    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-credential-socket\") pod \"08995227-51dd-4ba8-9bbe-c4bd9877903e\" (UID: \"08995227-51dd-4ba8-9bbe-c4bd9877903e\") "
Oct 30 21:22:12.547252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547247    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-kube-api-access-8fpsp\" (UniqueName: \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-kube-api-access-8fpsp\") pod \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\" (UID: \"ae1c5bd7-0550-4b20-9fa6-e5275b88253c\") "
Oct 30 21:22:12.547603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546646    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-credential-socket" pod "ae1c5bd7-0550-4b20-9fa6-e5275b88253c" (UID: "ae1c5bd7-0550-4b20-9fa6-e5275b88253c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:12.548135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.546728    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-data" pod "ae1c5bd7-0550-4b20-9fa6-e5275b88253c" (UID: "ae1c5bd7-0550-4b20-9fa6-e5275b88253c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:12.548135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547187    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08995227-51dd-4ba8-9bbe-c4bd9877903e-istiod-ca-cert" pod "08995227-51dd-4ba8-9bbe-c4bd9877903e" (UID: "08995227-51dd-4ba8-9bbe-c4bd9877903e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:12.548135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547565    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-certs" pod "ae1c5bd7-0550-4b20-9fa6-e5275b88253c" (UID: "ae1c5bd7-0550-4b20-9fa6-e5275b88253c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:12.548135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547575    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-certs" pod "08995227-51dd-4ba8-9bbe-c4bd9877903e" (UID: "08995227-51dd-4ba8-9bbe-c4bd9877903e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:12.548135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547584    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-socket" pod "ae1c5bd7-0550-4b20-9fa6-e5275b88253c" (UID: "ae1c5bd7-0550-4b20-9fa6-e5275b88253c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:12.548135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547945    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istiod-ca-cert" pod "ae1c5bd7-0550-4b20-9fa6-e5275b88253c" (UID: "ae1c5bd7-0550-4b20-9fa6-e5275b88253c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:12.548135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.547977    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-data" pod "08995227-51dd-4ba8-9bbe-c4bd9877903e" (UID: "08995227-51dd-4ba8-9bbe-c4bd9877903e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:12.548135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.548073    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.548653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.548212    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-credential-socket" pod "08995227-51dd-4ba8-9bbe-c4bd9877903e" (UID: "08995227-51dd-4ba8-9bbe-c4bd9877903e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:12.549366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.549331    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-envoy" pod "08995227-51dd-4ba8-9bbe-c4bd9877903e" (UID: "08995227-51dd-4ba8-9bbe-c4bd9877903e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:12.549514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.549346    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-podinfo" pod "08995227-51dd-4ba8-9bbe-c4bd9877903e" (UID: "08995227-51dd-4ba8-9bbe-c4bd9877903e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:12.549821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.549790    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-kube-api-access-gp97l" pod "08995227-51dd-4ba8-9bbe-c4bd9877903e" (UID: "08995227-51dd-4ba8-9bbe-c4bd9877903e"). InnerVolumeSpecName "kube-api-access-gp97l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:12.550223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.550204    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-token" pod "ae1c5bd7-0550-4b20-9fa6-e5275b88253c" (UID: "ae1c5bd7-0550-4b20-9fa6-e5275b88253c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:12.550284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.550232    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-token" pod "08995227-51dd-4ba8-9bbe-c4bd9877903e" (UID: "08995227-51dd-4ba8-9bbe-c4bd9877903e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:12.550523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.550502    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-podinfo" pod "ae1c5bd7-0550-4b20-9fa6-e5275b88253c" (UID: "ae1c5bd7-0550-4b20-9fa6-e5275b88253c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:12.550844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.550829    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-kube-api-access-8fpsp" pod "ae1c5bd7-0550-4b20-9fa6-e5275b88253c" (UID: "ae1c5bd7-0550-4b20-9fa6-e5275b88253c"). InnerVolumeSpecName "kube-api-access-8fpsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:12.551087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.551072    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-envoy" pod "ae1c5bd7-0550-4b20-9fa6-e5275b88253c" (UID: "ae1c5bd7-0550-4b20-9fa6-e5275b88253c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:12.588429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.588395    4810 generic.go:356] "Generic (PLEG): container finished" podID="08995227-51dd-4ba8-9bbe-c4bd9877903e" containerID="01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41" exitCode=0
Oct 30 21:22:12.588588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.588439    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"
Oct 30 21:22:12.588588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.588471    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" event={"ID":"08995227-51dd-4ba8-9bbe-c4bd9877903e","Type":"ContainerDied","Data":"01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41"}
Oct 30 21:22:12.588588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.588533    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr" event={"ID":"08995227-51dd-4ba8-9bbe-c4bd9877903e","Type":"ContainerDied","Data":"c977f065fb7c8ce34af7662526fd83b96659eb619eb30adc39a30313a83ba88b"}
Oct 30 21:22:12.588588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.588556    4810 scope.go:122] "RemoveContainer" containerID="01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41"
Oct 30 21:22:12.595730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.595703    4810 generic.go:356] "Generic (PLEG): container finished" podID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" containerID="3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e" exitCode=0
Oct 30 21:22:12.595902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.595752    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"
Oct 30 21:22:12.595902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.595777    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" event={"ID":"ae1c5bd7-0550-4b20-9fa6-e5275b88253c","Type":"ContainerDied","Data":"3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e"}
Oct 30 21:22:12.595902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.595823    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c" event={"ID":"ae1c5bd7-0550-4b20-9fa6-e5275b88253c","Type":"ContainerDied","Data":"2264eb2bb51f7ad9e2edf0f409c54a6a2e835923b2ecb771c12d53a0e294c359"}
Oct 30 21:22:12.602536 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.602519    4810 scope.go:122] "RemoveContainer" containerID="01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41"
Oct 30 21:22:12.602800 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:12.602779    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41\": container with ID starting with 01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41 not found: ID does not exist" containerID="01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41"
Oct 30 21:22:12.602933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.602809    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41"} err="failed to get container status \"01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41\": rpc error: code = NotFound desc = could not find container \"01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41\": container with ID starting with 01b381207e5925ce011aca871d63db0ae2fd58e7f3daaf7ed068db1469d4bf41 not found: ID does not exist"
Oct 30 21:22:12.602933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.602829    4810 scope.go:122] "RemoveContainer" containerID="3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e"
Oct 30 21:22:12.616820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.616793    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"]
Oct 30 21:22:12.617102 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.617083    4810 scope.go:122] "RemoveContainer" containerID="3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e"
Oct 30 21:22:12.617420 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:12.617400    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e\": container with ID starting with 3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e not found: ID does not exist" containerID="3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e"
Oct 30 21:22:12.617494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.617428    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e"} err="failed to get container status \"3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e\": rpc error: code = NotFound desc = could not find container \"3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e\": container with ID starting with 3c61c3c63dfbec0c44d2762a99a86f0adafa306e0869ff0e33d5337b04c68b4e not found: ID does not exist"
Oct 30 21:22:12.622240 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.622219    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-tv8dr"]
Oct 30 21:22:12.631132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.631109    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"]
Oct 30 21:22:12.637509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.637459    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-qqj4c"]
Oct 30 21:22:12.649132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649107    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649134    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649148    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649161    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649175    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649188    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649202    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649215    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649230    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649243    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649256    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649269    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649295    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08995227-51dd-4ba8-9bbe-c4bd9877903e-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649311    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fpsp\" (UniqueName: \"kubernetes.io/projected/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-kube-api-access-8fpsp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649326    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp97l\" (UniqueName: \"kubernetes.io/projected/08995227-51dd-4ba8-9bbe-c4bd9877903e-kube-api-access-gp97l\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649341    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1c5bd7-0550-4b20-9fa6-e5275b88253c-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:12.649597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:12.649355    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08995227-51dd-4ba8-9bbe-c4bd9877903e-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:13.435077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:13.435048    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:22:13.489200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:13.489162    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"]
Oct 30 21:22:13.664148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:13.664116    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08995227-51dd-4ba8-9bbe-c4bd9877903e" path="/var/lib/kubelet/pods/08995227-51dd-4ba8-9bbe-c4bd9877903e/volumes"
Oct 30 21:22:13.664718 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:13.664697    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" path="/var/lib/kubelet/pods/ae1c5bd7-0550-4b20-9fa6-e5275b88253c/volumes"
Oct 30 21:22:14.774015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.773975    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:22:14.811227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.811197    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:22:14.965617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.965582    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"]
Oct 30 21:22:14.966377 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.966360    4810 cpu_manager.go:414] "containerMap: removing container" podUID="08995227-51dd-4ba8-9bbe-c4bd9877903e" containerName="istio-proxy"
Oct 30 21:22:14.966377 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.966376    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08995227-51dd-4ba8-9bbe-c4bd9877903e" containerName="istio-proxy"
Oct 30 21:22:14.966567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.966393    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" containerName="istio-proxy"
Oct 30 21:22:14.966567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.966398    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" containerName="istio-proxy"
Oct 30 21:22:14.966567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.966520    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="08995227-51dd-4ba8-9bbe-c4bd9877903e" containerName="istio-proxy"
Oct 30 21:22:14.966567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.966530    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1c5bd7-0550-4b20-9fa6-e5275b88253c" containerName="istio-proxy"
Oct 30 21:22:14.972400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.972373    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:14.982050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.982024    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"]
Oct 30 21:22:14.982449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.982430    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"agc-cred-f1750c1e692f\""
Oct 30 21:22:14.984022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.983395    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"openshift-service-ca.crt\""
Oct 30 21:22:14.984022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.983412    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"routing-overlay\""
Oct 30 21:22:14.984022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.983457    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6741b1\""
Oct 30 21:22:14.984022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:14.983566    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6741b1\"/\"kube-root-ca.crt\""
Oct 30 21:22:15.099810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.099712    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.100014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.099829    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzf9\" (UniqueName: \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-kube-api-access-7qzf9\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.100014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.099979    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.100014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.100005    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.204909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.203894    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.204909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.204562    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzf9\" (UniqueName: \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-kube-api-access-7qzf9\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.204909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.204712    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.204909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.204743    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.207909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.205750    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.222884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.219055    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.236977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.233705    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.236977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.236915    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzf9\" (UniqueName: \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-kube-api-access-7qzf9\") pod \"payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") " pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.304383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.304111    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:15.661807 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:22:15.661775    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8065b8_f58e_42c0_8d6d_8718fee7446c.slice/crio-c53dcf45d66a73ced148908fc1c139b748bdc3fd566ecfdba75d68c94fdf4fcc WatchSource:0}: Error finding container c53dcf45d66a73ced148908fc1c139b748bdc3fd566ecfdba75d68c94fdf4fcc: Status 404 returned error can't find the container with id c53dcf45d66a73ced148908fc1c139b748bdc3fd566ecfdba75d68c94fdf4fcc
Oct 30 21:22:15.668572 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:15.668547    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"]
Oct 30 21:22:16.011435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.011360    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"]
Oct 30 21:22:16.016144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.016119    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.021383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.021359    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"agc-cred-af899b90f8ba\""
Oct 30 21:22:16.021543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.021474    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"kube-root-ca.crt\""
Oct 30 21:22:16.021543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.021538    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"openshift-service-ca.crt\""
Oct 30 21:22:16.021665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.021596    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d95261\""
Oct 30 21:22:16.021725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.021683    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"routing-overlay\""
Oct 30 21:22:16.022314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.021777    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d95261\"/\"payload-processing-external-model-e2e--8ac6fff3-dockercfg-qn2ph\""
Oct 30 21:22:16.023902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.023881    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"]
Oct 30 21:22:16.136413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.136378    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhn4\" (UniqueName: \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-kube-api-access-pzhn4\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.136607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.136586    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.136667 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.136629    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.136706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.136686    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.256005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.248244    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhn4\" (UniqueName: \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-kube-api-access-pzhn4\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.256005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.249651    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.256005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.249741    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.256005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.249795    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.256005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.251618    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.256005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.252938    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.279921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.276229    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.301880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.288665    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhn4\" (UniqueName: \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-kube-api-access-pzhn4\") pod \"payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") " pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.336878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.336110    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:16.634236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.634200    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"]
Oct 30 21:22:16.637985 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:22:16.637959    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1388555d_21ed_4e4f_a201_307499a352dc.slice/crio-4aa4f5724913fe673b7dc047ec0ac4f926471c0c4a051c7d09bb208fe819bfee WatchSource:0}: Error finding container 4aa4f5724913fe673b7dc047ec0ac4f926471c0c4a051c7d09bb208fe819bfee: Status 404 returned error can't find the container with id 4aa4f5724913fe673b7dc047ec0ac4f926471c0c4a051c7d09bb208fe819bfee
Oct 30 21:22:16.646468 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.646442    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg" event={"ID":"cc8065b8-f58e-42c0-8d6d-8718fee7446c","Type":"ContainerStarted","Data":"d701dc2bc30306ade2ba2af98a39a3b8598662c2324e800904fae9c6852bf54f"}
Oct 30 21:22:16.646548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.646504    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:16.646548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.646522    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg" event={"ID":"cc8065b8-f58e-42c0-8d6d-8718fee7446c","Type":"ContainerStarted","Data":"c53dcf45d66a73ced148908fc1c139b748bdc3fd566ecfdba75d68c94fdf4fcc"}
Oct 30 21:22:16.664038 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:16.663995    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg" podStartSLOduration=2.663983485 podStartE2EDuration="2.663983485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:16.661766478 +0000 UTC m=+2703.153583024" watchObservedRunningTime="2026-10-30 21:22:16.663983485 +0000 UTC m=+2703.155800029"
Oct 30 21:22:17.663785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:17.663752    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:17.664214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:17.663826    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5" event={"ID":"1388555d-21ed-4e4f-a201-307499a352dc","Type":"ContainerStarted","Data":"1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600"}
Oct 30 21:22:17.664214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:17.663845    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5" event={"ID":"1388555d-21ed-4e4f-a201-307499a352dc","Type":"ContainerStarted","Data":"4aa4f5724913fe673b7dc047ec0ac4f926471c0c4a051c7d09bb208fe819bfee"}
Oct 30 21:22:17.677973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:17.677919    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5" podStartSLOduration=2.677901785 podStartE2EDuration="2.677901785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:17.671502128 +0000 UTC m=+2704.163318673" watchObservedRunningTime="2026-10-30 21:22:17.677901785 +0000 UTC m=+2704.169718330"
Oct 30 21:22:17.972084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:17.972006    4810 scope.go:122] "RemoveContainer" containerID="cdaa41483bbb6990ee348932268320f43aa960154e23018b400897b2565198b4"
Oct 30 21:22:17.986655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:17.986625    4810 scope.go:122] "RemoveContainer" containerID="fc8663455d2b5dce4df22c373b0922fc2270b8c7453e605e058ba7b30f079110"
Oct 30 21:22:17.999085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:17.999057    4810 scope.go:122] "RemoveContainer" containerID="522517c01ffdb24e884d4314e7c7ee1cbfa3cb1adaf26b37b2869c51ef6e8d51"
Oct 30 21:22:18.012255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.012230    4810 scope.go:122] "RemoveContainer" containerID="08041c791c3816fdacacf6db174f6905fac2e76c70ca8cfb4f60b87949917cbd"
Oct 30 21:22:18.023031 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.022993    4810 scope.go:122] "RemoveContainer" containerID="c26c641d19b9a065e85b95fa0942e1fce0749c5192969af7940449851cb9c586"
Oct 30 21:22:18.032464 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.032435    4810 scope.go:122] "RemoveContainer" containerID="b55bdc53045775fc58fa32d75d1c903973f7cc891b05ebf174c34c2ab6023b05"
Oct 30 21:22:18.041844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.041824    4810 scope.go:122] "RemoveContainer" containerID="2d33dded35176d5ebdaaaf42ed9ea4a78289fd5b98bc3f9ec9bf544398793437"
Oct 30 21:22:18.051301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.051286    4810 scope.go:122] "RemoveContainer" containerID="3001e0ddebd6c5dd05257113ce8840c74de9f2fc1e97bf9f23eb7e6d015cab99"
Oct 30 21:22:18.060122 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.060107    4810 scope.go:122] "RemoveContainer" containerID="13d6e89109bdb50f0643357c1e5c881b190b88b1adcf524df09eecc8a4bc2e1c"
Oct 30 21:22:18.490552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.490461    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d" podUID="64069fea-0ef4-4e09-987c-0ebc2c9d8298" containerName="payload-processing" containerID="cri-o://83b07498c680d1823e479dbd377fce61956e4d96c2de8b2c9711867af6d5f392" gracePeriod=25
Oct 30 21:22:18.665514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.665482    4810 generic.go:356] "Generic (PLEG): container finished" podID="64069fea-0ef4-4e09-987c-0ebc2c9d8298" containerID="83b07498c680d1823e479dbd377fce61956e4d96c2de8b2c9711867af6d5f392" exitCode=0
Oct 30 21:22:18.666024 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.665556    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d" event={"ID":"64069fea-0ef4-4e09-987c-0ebc2c9d8298","Type":"ContainerDied","Data":"83b07498c680d1823e479dbd377fce61956e4d96c2de8b2c9711867af6d5f392"}
Oct 30 21:22:18.835168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.835138    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:22:18.917986 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.917951    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-provider-credentials\" (UniqueName: \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-provider-credentials\") pod \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") "
Oct 30 21:22:18.918185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.918071    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-kube-api-access-4mhdl\" (UniqueName: \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-kube-api-access-4mhdl\") pod \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") "
Oct 30 21:22:18.918185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.918126    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-routing-overlay-volume\") pod \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") "
Oct 30 21:22:18.918309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.918216    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-plugins-config-volume\") pod \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\" (UID: \"64069fea-0ef4-4e09-987c-0ebc2c9d8298\") "
Oct 30 21:22:18.918584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.918548    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-routing-overlay-volume" pod "64069fea-0ef4-4e09-987c-0ebc2c9d8298" (UID: "64069fea-0ef4-4e09-987c-0ebc2c9d8298"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:18.918584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.918574    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-plugins-config-volume" pod "64069fea-0ef4-4e09-987c-0ebc2c9d8298" (UID: "64069fea-0ef4-4e09-987c-0ebc2c9d8298"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:18.918852 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.918825    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:18.918986 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.918853    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64069fea-0ef4-4e09-987c-0ebc2c9d8298-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:18.920287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.920268    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-kube-api-access-4mhdl" pod "64069fea-0ef4-4e09-987c-0ebc2c9d8298" (UID: "64069fea-0ef4-4e09-987c-0ebc2c9d8298"). InnerVolumeSpecName "kube-api-access-4mhdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:18.920604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:18.920582    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-provider-credentials" pod "64069fea-0ef4-4e09-987c-0ebc2c9d8298" (UID: "64069fea-0ef4-4e09-987c-0ebc2c9d8298"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:19.020409 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:19.020334    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:19.020409 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:19.020363    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhdl\" (UniqueName: \"kubernetes.io/projected/64069fea-0ef4-4e09-987c-0ebc2c9d8298-kube-api-access-4mhdl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:19.675932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:19.675892    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d" event={"ID":"64069fea-0ef4-4e09-987c-0ebc2c9d8298","Type":"ContainerDied","Data":"759c6fae3352938142e00b7a2f0f848623366960150f64b80c0d54ae075a23a1"}
Oct 30 21:22:19.675932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:19.675919    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"
Oct 30 21:22:19.676412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:19.675968    4810 scope.go:122] "RemoveContainer" containerID="83b07498c680d1823e479dbd377fce61956e4d96c2de8b2c9711867af6d5f392"
Oct 30 21:22:19.698096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:19.698064    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"]
Oct 30 21:22:19.702946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:19.702926    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4fdd8ddc-ctw4d"]
Oct 30 21:22:20.067747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:20.067658    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"]
Oct 30 21:22:20.067951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:20.067885    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy" containerID="cri-o://9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538" gracePeriod=30
Oct 30 21:22:21.665154 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:21.665118    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64069fea-0ef4-4e09-987c-0ebc2c9d8298" path="/var/lib/kubelet/pods/64069fea-0ef4-4e09-987c-0ebc2c9d8298/volumes"
Oct 30 21:22:25.162326 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:25.162282    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:22:25.420345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.420318    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 21:22:25.530878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.530695    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-certs\") pod \"168be12e-db11-4849-99e4-290d38a4e8ed\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") "
Oct 30 21:22:25.535716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.531007    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-certs" pod "168be12e-db11-4849-99e4-290d38a4e8ed" (UID: "168be12e-db11-4849-99e4-290d38a4e8ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:25.535716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.531391    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-data" pod "168be12e-db11-4849-99e4-290d38a4e8ed" (UID: "168be12e-db11-4849-99e4-290d38a4e8ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:25.535716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.531170    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-data\") pod \"168be12e-db11-4849-99e4-290d38a4e8ed\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") "
Oct 30 21:22:25.535716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.531521    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-istio-token\" (UniqueName: \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-istio-token\") pod \"168be12e-db11-4849-99e4-290d38a4e8ed\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") "
Oct 30 21:22:25.535716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.532059    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-socket\") pod \"168be12e-db11-4849-99e4-290d38a4e8ed\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") "
Oct 30 21:22:25.535716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.532313    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-socket" pod "168be12e-db11-4849-99e4-290d38a4e8ed" (UID: "168be12e-db11-4849-99e4-290d38a4e8ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:25.535716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.532373    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-credential-socket\") pod \"168be12e-db11-4849-99e4-290d38a4e8ed\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") "
Oct 30 21:22:25.535716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.532591    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-credential-socket" pod "168be12e-db11-4849-99e4-290d38a4e8ed" (UID: "168be12e-db11-4849-99e4-290d38a4e8ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:25.535716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.532650    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168be12e-db11-4849-99e4-290d38a4e8ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168be12e-db11-4849-99e4-290d38a4e8ed-istiod-ca-cert\") pod \"168be12e-db11-4849-99e4-290d38a4e8ed\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") "
Oct 30 21:22:25.535716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.532999    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168be12e-db11-4849-99e4-290d38a4e8ed-istiod-ca-cert" pod "168be12e-db11-4849-99e4-290d38a4e8ed" (UID: "168be12e-db11-4849-99e4-290d38a4e8ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:25.536389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.533057    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-envoy\") pod \"168be12e-db11-4849-99e4-290d38a4e8ed\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") "
Oct 30 21:22:25.536389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.536160    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-kube-api-access-x5c72\" (UniqueName: \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-kube-api-access-x5c72\") pod \"168be12e-db11-4849-99e4-290d38a4e8ed\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") "
Oct 30 21:22:25.536389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.536250    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/168be12e-db11-4849-99e4-290d38a4e8ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168be12e-db11-4849-99e4-290d38a4e8ed-istio-podinfo\") pod \"168be12e-db11-4849-99e4-290d38a4e8ed\" (UID: \"168be12e-db11-4849-99e4-290d38a4e8ed\") "
Oct 30 21:22:25.552898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.536943    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:25.552898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.536970    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:25.552898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.536987    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168be12e-db11-4849-99e4-290d38a4e8ed-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:25.552898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.537004    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:25.552898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.537019    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:25.552898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.545199    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/168be12e-db11-4849-99e4-290d38a4e8ed-istio-podinfo" pod "168be12e-db11-4849-99e4-290d38a4e8ed" (UID: "168be12e-db11-4849-99e4-290d38a4e8ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:25.557238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.557034    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-istio-token" pod "168be12e-db11-4849-99e4-290d38a4e8ed" (UID: "168be12e-db11-4849-99e4-290d38a4e8ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:25.559138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.559043    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-envoy" pod "168be12e-db11-4849-99e4-290d38a4e8ed" (UID: "168be12e-db11-4849-99e4-290d38a4e8ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:25.559138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.559080    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-kube-api-access-x5c72" pod "168be12e-db11-4849-99e4-290d38a4e8ed" (UID: "168be12e-db11-4849-99e4-290d38a4e8ed"). InnerVolumeSpecName "kube-api-access-x5c72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:25.641561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.641434    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:25.641561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.641466    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168be12e-db11-4849-99e4-290d38a4e8ed-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:25.641561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.641483    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5c72\" (UniqueName: \"kubernetes.io/projected/168be12e-db11-4849-99e4-290d38a4e8ed-kube-api-access-x5c72\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:25.641561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.641498    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168be12e-db11-4849-99e4-290d38a4e8ed-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:25.735451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.735416    4810 generic.go:356] "Generic (PLEG): container finished" podID="168be12e-db11-4849-99e4-290d38a4e8ed" containerID="9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538" exitCode=0
Oct 30 21:22:25.735630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.735609    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" event={"ID":"168be12e-db11-4849-99e4-290d38a4e8ed","Type":"ContainerDied","Data":"9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538"}
Oct 30 21:22:25.735713 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.735669    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" event={"ID":"168be12e-db11-4849-99e4-290d38a4e8ed","Type":"ContainerDied","Data":"3e9eaa033e4837691afe9050ea906a10137482720a09aed9295b6c4b280174ec"}
Oct 30 21:22:25.735713 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.735700    4810 scope.go:122] "RemoveContainer" containerID="9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538"
Oct 30 21:22:25.735871 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.735846    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"
Oct 30 21:22:25.764093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.760737    4810 scope.go:122] "RemoveContainer" containerID="9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538"
Oct 30 21:22:25.764093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.760629    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"]
Oct 30 21:22:25.764093 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:25.761188    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538\": container with ID starting with 9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538 not found: ID does not exist" containerID="9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538"
Oct 30 21:22:25.764093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.761223    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538"} err="failed to get container status \"9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538\": rpc error: code = NotFound desc = could not find container \"9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538\": container with ID starting with 9c78d24621f4ad84faaa7a43a227706d4e9dca6961df557251048c7fd4985538 not found: ID does not exist"
Oct 30 21:22:25.767828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:25.767803    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2"]
Oct 30 21:22:26.400882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:26.399934    4810 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:22:26.400882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:26.400017    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-v4rv2" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:22:27.668164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:27.668113    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" path="/var/lib/kubelet/pods/168be12e-db11-4849-99e4-290d38a4e8ed/volumes"
Oct 30 21:22:27.669143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:27.669112    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:22:28.669698 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:28.669668    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:22:41.653116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:41.653084    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:22:47.576258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.576220    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"]
Oct 30 21:22:47.577215 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.577192    4810 cpu_manager.go:414] "containerMap: removing container" podUID="64069fea-0ef4-4e09-987c-0ebc2c9d8298" containerName="payload-processing"
Oct 30 21:22:47.577215 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.577215    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64069fea-0ef4-4e09-987c-0ebc2c9d8298" containerName="payload-processing"
Oct 30 21:22:47.577404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.577239    4810 cpu_manager.go:414] "containerMap: removing container" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy"
Oct 30 21:22:47.577404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.577246    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy"
Oct 30 21:22:47.577519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.577492    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="168be12e-db11-4849-99e4-290d38a4e8ed" containerName="istio-proxy"
Oct 30 21:22:47.577581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.577526    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="64069fea-0ef4-4e09-987c-0ebc2c9d8298" containerName="payload-processing"
Oct 30 21:22:47.581577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.581554    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.586840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.586813    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99668e\"/\"agc-cred-c4555bdbddb7\""
Oct 30 21:22:47.588991 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.588971    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"]
Oct 30 21:22:47.746681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.746288    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9jh6\" (UniqueName: \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-kube-api-access-n9jh6\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.747153 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.747127    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.747359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.747342    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.747518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.747499    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.849243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.849122    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.849243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.849188    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.849243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.849210    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.850994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.849267    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9jh6\" (UniqueName: \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-kube-api-access-n9jh6\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.850994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.850498    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.850994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.850674    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.852584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.852557    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.854630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.854605    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"]
Oct 30 21:22:47.858636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.858612    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9jh6\" (UniqueName: \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-kube-api-access-n9jh6\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.859811 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.859794    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:47.868613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.868585    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"]
Oct 30 21:22:47.868722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.868665    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e07442\"/\"agc-cred-ba58a54759e0\""
Oct 30 21:22:47.898527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.898502    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:47.979876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.974003    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:47.979876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.974093    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:47.979876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.974277    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4hz\" (UniqueName: \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-kube-api-access-kn4hz\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:47.979876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:47.974307    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:48.076351 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.076307    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:48.076555 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.076399    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:48.076633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.076572    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4hz\" (UniqueName: \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-kube-api-access-kn4hz\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:48.076633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.076603    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:48.077765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.077254    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:48.077765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.077718    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:48.081320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.081272    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:48.086318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.086298    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4hz\" (UniqueName: \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-kube-api-access-kn4hz\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:48.207475 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.207419    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:48.233351 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.233309    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"]
Oct 30 21:22:48.234943 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:22:48.234917    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a039b8_55d3_40eb_966a_b94255e14436.slice/crio-9c3c0dbd23d99aa5b4372f591923f8a1c610d0f4107d03d37a89c5605a905fb9 WatchSource:0}: Error finding container 9c3c0dbd23d99aa5b4372f591923f8a1c610d0f4107d03d37a89c5605a905fb9: Status 404 returned error can't find the container with id 9c3c0dbd23d99aa5b4372f591923f8a1c610d0f4107d03d37a89c5605a905fb9
Oct 30 21:22:48.553931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.553374    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"]
Oct 30 21:22:48.567891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.565832    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.583885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.575849    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e8063c\"/\"agc-cred-d169d06edf8a\""
Oct 30 21:22:48.596399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.596172    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"]
Oct 30 21:22:48.609897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.607347    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"]
Oct 30 21:22:48.695647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.695610    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.695841 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.695819    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxpcg\" (UniqueName: \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-kube-api-access-gxpcg\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.695949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.695914    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.696024 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.695966    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.796948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.796912    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxpcg\" (UniqueName: \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-kube-api-access-gxpcg\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.797098 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.796996    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.797098 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.797060    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.797224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.797162    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.797764 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.797738    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.797914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.797893    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.800138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.800122    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.816481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.816424    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxpcg\" (UniqueName: \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-kube-api-access-gxpcg\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:48.913343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:48.913082    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:49.009567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:49.008478    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv" event={"ID":"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d","Type":"ContainerStarted","Data":"03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c"}
Oct 30 21:22:49.009567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:49.008545    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv" event={"ID":"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d","Type":"ContainerStarted","Data":"0ee8336c301157121b4a9097af979dc48755b61b440a75bfe329d358dfd8af61"}
Oct 30 21:22:49.009567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:49.009503    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:49.013535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:49.012009    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75" event={"ID":"93a039b8-55d3-40eb-966a-b94255e14436","Type":"ContainerStarted","Data":"3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9"}
Oct 30 21:22:49.013535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:49.012059    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75" event={"ID":"93a039b8-55d3-40eb-966a-b94255e14436","Type":"ContainerStarted","Data":"9c3c0dbd23d99aa5b4372f591923f8a1c610d0f4107d03d37a89c5605a905fb9"}
Oct 30 21:22:49.013535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:49.012673    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:49.035310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:49.035263    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv" podStartSLOduration=2.035234165 podStartE2EDuration="2.035234165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:49.028745665 +0000 UTC m=+2735.520562211" watchObservedRunningTime="2026-10-30 21:22:49.035234165 +0000 UTC m=+2735.527050711"
Oct 30 21:22:49.065999 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:49.053685    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75" podStartSLOduration=2.053667429 podStartE2EDuration="2.053667429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:49.048263307 +0000 UTC m=+2735.540079854" watchObservedRunningTime="2026-10-30 21:22:49.053667429 +0000 UTC m=+2735.545483974"
Oct 30 21:22:49.244044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:49.244014    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"]
Oct 30 21:22:49.244580 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:22:49.244554    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3257db_84f2_49b7_b136_73a5cc11e862.slice/crio-5087e6ed87fde26caa940a051fc2dc8cbe42513039b10abb9fac0b42be8815cc WatchSource:0}: Error finding container 5087e6ed87fde26caa940a051fc2dc8cbe42513039b10abb9fac0b42be8815cc: Status 404 returned error can't find the container with id 5087e6ed87fde26caa940a051fc2dc8cbe42513039b10abb9fac0b42be8815cc
Oct 30 21:22:50.023023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.022917    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls" event={"ID":"3a3257db-84f2-49b7-b136-73a5cc11e862","Type":"ContainerStarted","Data":"27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082"}
Oct 30 21:22:50.023023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.022985    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls" event={"ID":"3a3257db-84f2-49b7-b136-73a5cc11e862","Type":"ContainerStarted","Data":"5087e6ed87fde26caa940a051fc2dc8cbe42513039b10abb9fac0b42be8815cc"}
Oct 30 21:22:50.023594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.023215    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:50.048671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.048619    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls" podStartSLOduration=2.048603847 podStartE2EDuration="2.048603847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:50.046199194 +0000 UTC m=+2736.538015741" watchObservedRunningTime="2026-10-30 21:22:50.048603847 +0000 UTC m=+2736.540420426"
Oct 30 21:22:50.059254 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.059224    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"]
Oct 30 21:22:50.064113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.064091    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.069106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.069083    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1e0f5\"/\"agc-cred-f721662d89dc\""
Oct 30 21:22:50.078615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.078591    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"]
Oct 30 21:22:50.123515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.123471    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.123721 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.123555    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e07442/agc-cred-ba58a54759e0: secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:50.123721 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.123579    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv: secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:50.123721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.123616    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.123721 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.123647    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials podName:41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d nodeName:}" failed. No retries permitted until 2026-10-30 21:22:50.623628723 +0000 UTC m=+2737.115445263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv" (UID: "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d") : secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:50.124025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.123748    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw7j\" (UniqueName: \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-kube-api-access-lqw7j\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.124025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.123786    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.125909 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.124962    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-c4555bdbddb7: secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:50.125909 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.124982    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75: secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:50.125909 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.125031    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials podName:93a039b8-55d3-40eb-966a-b94255e14436 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:50.625016325 +0000 UTC m=+2737.116832863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75" (UID: "93a039b8-55d3-40eb-966a-b94255e14436") : secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:50.226669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.226631    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw7j\" (UniqueName: \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-kube-api-access-lqw7j\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.226889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.226688    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.226986 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.226967    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.227063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.227005    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.227588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.227555    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.228186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.228165    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.229606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.229584    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.235581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.235555    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw7j\" (UniqueName: \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-kube-api-access-lqw7j\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.353474 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.352581    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"]
Oct 30 21:22:50.394446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.394407    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:50.640313 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.640281    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e07442/agc-cred-ba58a54759e0: secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:50.640313 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.640306    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv: secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:50.640589 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.640382    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials podName:41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d nodeName:}" failed. No retries permitted until 2026-10-30 21:22:51.640344172 +0000 UTC m=+2738.132160709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv" (UID: "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d") : secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:50.640589 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.640393    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-c4555bdbddb7: secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:50.640589 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.640414    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75: secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:50.640589 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:50.640469    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials podName:93a039b8-55d3-40eb-966a-b94255e14436 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:51.640451618 +0000 UTC m=+2738.132268141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75" (UID: "93a039b8-55d3-40eb-966a-b94255e14436") : secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:50.685701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.685665    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"]
Oct 30 21:22:50.686617 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:22:50.686588    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066f80a8_e01a_465d_8dae_b6cfbaa4128a.slice/crio-feebf4ed0cbecd46b194db84a83a15b58cf270dab03ed4be1adec694feed73c3 WatchSource:0}: Error finding container feebf4ed0cbecd46b194db84a83a15b58cf270dab03ed4be1adec694feed73c3: Status 404 returned error can't find the container with id feebf4ed0cbecd46b194db84a83a15b58cf270dab03ed4be1adec694feed73c3
Oct 30 21:22:50.733691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.733667    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"]
Oct 30 21:22:50.797185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.797157    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"]
Oct 30 21:22:50.801797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.801780    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:50.806892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.805401    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:50.806892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.805615    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:22:50.809464 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.809436    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"]
Oct 30 21:22:50.843138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.843107    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxc6v\" (UniqueName: \"kubernetes.io/projected/a4bbbba3-7b21-4172-8154-b930de12ad0f-kube-api-access-xxc6v\") pod \"maas-api-revoke-sub-1a163532-dbdjw\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:50.843383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.843362    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a163532-dbdjw\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:50.843651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.843480    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a163532-dbdjw\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:50.969193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.969100    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a163532-dbdjw\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:50.969517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.969494    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a163532-dbdjw\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:50.977415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.977385    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxc6v\" (UniqueName: \"kubernetes.io/projected/a4bbbba3-7b21-4172-8154-b930de12ad0f-kube-api-access-xxc6v\") pod \"maas-api-revoke-sub-1a163532-dbdjw\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:50.978558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.976571    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a163532-dbdjw\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:50.991680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.991647    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a163532-dbdjw\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:51.002400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:50.993633    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxc6v\" (UniqueName: \"kubernetes.io/projected/a4bbbba3-7b21-4172-8154-b930de12ad0f-kube-api-access-xxc6v\") pod \"maas-api-revoke-sub-1a163532-dbdjw\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:51.032333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:51.032291    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk" event={"ID":"066f80a8-e01a-465d-8dae-b6cfbaa4128a","Type":"ContainerStarted","Data":"9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b"}
Oct 30 21:22:51.032784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:51.032510    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:51.032784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:51.032556    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk" event={"ID":"066f80a8-e01a-465d-8dae-b6cfbaa4128a","Type":"ContainerStarted","Data":"feebf4ed0cbecd46b194db84a83a15b58cf270dab03ed4be1adec694feed73c3"}
Oct 30 21:22:51.051975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:51.051927    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk" podStartSLOduration=1.051913984 podStartE2EDuration="1.051913984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:51.049177194 +0000 UTC m=+2737.540993739" watchObservedRunningTime="2026-10-30 21:22:51.051913984 +0000 UTC m=+2737.543730569"
Oct 30 21:22:51.125495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:51.125462    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:51.410316 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:22:51.410269    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bbbba3_7b21_4172_8154_b930de12ad0f.slice/crio-c86076b26bc8454d9a009ca92dc957150df1b9c67ea11ef4e8d321c3a236dd85 WatchSource:0}: Error finding container c86076b26bc8454d9a009ca92dc957150df1b9c67ea11ef4e8d321c3a236dd85: Status 404 returned error can't find the container with id c86076b26bc8454d9a009ca92dc957150df1b9c67ea11ef4e8d321c3a236dd85
Oct 30 21:22:51.411648 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:51.411609    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"]
Oct 30 21:22:51.695537 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:51.695500    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-c4555bdbddb7: secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:51.695537 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:51.695524    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75: secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:51.695537 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:51.695527    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e07442/agc-cred-ba58a54759e0: secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:51.695772 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:51.695549    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv: secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:51.695772 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:51.695571    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials podName:93a039b8-55d3-40eb-966a-b94255e14436 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:53.69555788 +0000 UTC m=+2740.187374420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75" (UID: "93a039b8-55d3-40eb-966a-b94255e14436") : secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:51.695772 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:51.695620    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials podName:41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d nodeName:}" failed. No retries permitted until 2026-10-30 21:22:53.695590742 +0000 UTC m=+2740.187407267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv" (UID: "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d") : secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:52.042233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:52.042191    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw" event={"ID":"a4bbbba3-7b21-4172-8154-b930de12ad0f","Type":"ContainerStarted","Data":"4bc25b170215f65f7195536595b778f30e37ac0ff28ad5529f0f7dd20bd31431"}
Oct 30 21:22:52.042655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:52.042266    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw" event={"ID":"a4bbbba3-7b21-4172-8154-b930de12ad0f","Type":"ContainerStarted","Data":"c86076b26bc8454d9a009ca92dc957150df1b9c67ea11ef4e8d321c3a236dd85"}
Oct 30 21:22:52.062008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:52.061955    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw" podStartSLOduration=2.061940119 podStartE2EDuration="2.061940119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:52.058211209 +0000 UTC m=+2738.550027754" watchObservedRunningTime="2026-10-30 21:22:52.061940119 +0000 UTC m=+2738.553756664"
Oct 30 21:22:53.052047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:53.051963    4810 generic.go:356] "Generic (PLEG): container finished" podID="a4bbbba3-7b21-4172-8154-b930de12ad0f" containerID="4bc25b170215f65f7195536595b778f30e37ac0ff28ad5529f0f7dd20bd31431" exitCode=0
Oct 30 21:22:53.052424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:53.052048    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw" event={"ID":"a4bbbba3-7b21-4172-8154-b930de12ad0f","Type":"ContainerDied","Data":"4bc25b170215f65f7195536595b778f30e37ac0ff28ad5529f0f7dd20bd31431"}
Oct 30 21:22:53.731427 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:53.731391    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-c4555bdbddb7: secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:53.731427 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:53.731422    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75: secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:53.731641 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:53.731439    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e07442/agc-cred-ba58a54759e0: secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:53.731641 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:53.731455    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv: secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:53.731641 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:53.731472    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials podName:93a039b8-55d3-40eb-966a-b94255e14436 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:57.731456104 +0000 UTC m=+2744.223272641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75" (UID: "93a039b8-55d3-40eb-966a-b94255e14436") : secret "agc-cred-c4555bdbddb7" not found
Oct 30 21:22:53.731641 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:53.731492    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials podName:41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d nodeName:}" failed. No retries permitted until 2026-10-30 21:22:57.731480474 +0000 UTC m=+2744.223297011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv" (UID: "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d") : secret "agc-cred-ba58a54759e0" not found
Oct 30 21:22:54.273030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.273004    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:54.443116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.443079    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bbbba3-7b21-4172-8154-b930de12ad0f-kube-api-access-xxc6v\" (UniqueName: \"kubernetes.io/projected/a4bbbba3-7b21-4172-8154-b930de12ad0f-kube-api-access-xxc6v\") pod \"a4bbbba3-7b21-4172-8154-b930de12ad0f\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") "
Oct 30 21:22:54.443312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.443124    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-xks-ca\") pod \"a4bbbba3-7b21-4172-8154-b930de12ad0f\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") "
Oct 30 21:22:54.443386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.443314    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-openshift-ca\") pod \"a4bbbba3-7b21-4172-8154-b930de12ad0f\" (UID: \"a4bbbba3-7b21-4172-8154-b930de12ad0f\") "
Oct 30 21:22:54.443623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.443601    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-openshift-ca" pod "a4bbbba3-7b21-4172-8154-b930de12ad0f" (UID: "a4bbbba3-7b21-4172-8154-b930de12ad0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:54.443837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.443821    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.445216 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.445193    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bbbba3-7b21-4172-8154-b930de12ad0f-kube-api-access-xxc6v" pod "a4bbbba3-7b21-4172-8154-b930de12ad0f" (UID: "a4bbbba3-7b21-4172-8154-b930de12ad0f"). InnerVolumeSpecName "kube-api-access-xxc6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:54.445216 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.445209    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-xks-ca" pod "a4bbbba3-7b21-4172-8154-b930de12ad0f" (UID: "a4bbbba3-7b21-4172-8154-b930de12ad0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:54.545690 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.545649    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxc6v\" (UniqueName: \"kubernetes.io/projected/a4bbbba3-7b21-4172-8154-b930de12ad0f-kube-api-access-xxc6v\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.545690 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.545690    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4bbbba3-7b21-4172-8154-b930de12ad0f-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.790069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.789895    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"]
Oct 30 21:22:54.819081 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:54.819045    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:22:55.067914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.067822    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"
Oct 30 21:22:55.067914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.067865    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw" event={"ID":"a4bbbba3-7b21-4172-8154-b930de12ad0f","Type":"ContainerDied","Data":"c86076b26bc8454d9a009ca92dc957150df1b9c67ea11ef4e8d321c3a236dd85"}
Oct 30 21:22:55.068104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.067917    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c86076b26bc8454d9a009ca92dc957150df1b9c67ea11ef4e8d321c3a236dd85"
Oct 30 21:22:55.158674 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:55.158634    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:22:55.188165 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.188132    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"]
Oct 30 21:22:55.191966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.191940    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:22:55.274994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.274965    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"]
Oct 30 21:22:55.276415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.276177    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a4bbbba3-7b21-4172-8154-b930de12ad0f" containerName="revoke-keys"
Oct 30 21:22:55.276415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.276199    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4bbbba3-7b21-4172-8154-b930de12ad0f" containerName="revoke-keys"
Oct 30 21:22:55.276557 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.276449    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4bbbba3-7b21-4172-8154-b930de12ad0f" containerName="revoke-keys"
Oct 30 21:22:55.280578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.280556    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.286614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.286590    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:22:55.294894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.291989    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:55.294894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.292615    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"]
Oct 30 21:22:55.470295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.470033    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf5mp\" (UniqueName: \"kubernetes.io/projected/11f441d2-d449-4a47-9ec9-2af6280e89bb-kube-api-access-wf5mp\") pod \"maas-api-revoke-sub-2ba5848c-ptfrj\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.470295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.470095    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba5848c-ptfrj\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.470295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.470146    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba5848c-ptfrj\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.573070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.573036    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf5mp\" (UniqueName: \"kubernetes.io/projected/11f441d2-d449-4a47-9ec9-2af6280e89bb-kube-api-access-wf5mp\") pod \"maas-api-revoke-sub-2ba5848c-ptfrj\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.573070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.573076    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba5848c-ptfrj\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.573344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.573110    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba5848c-ptfrj\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.574062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.574026    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba5848c-ptfrj\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.576014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.575986    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba5848c-ptfrj\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.581760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.581736    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf5mp\" (UniqueName: \"kubernetes.io/projected/11f441d2-d449-4a47-9ec9-2af6280e89bb-kube-api-access-wf5mp\") pod \"maas-api-revoke-sub-2ba5848c-ptfrj\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.622040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.622015    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:55.920612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:55.920579    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"]
Oct 30 21:22:55.922072 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:22:55.922048    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f441d2_d449_4a47_9ec9_2af6280e89bb.slice/crio-73e9ff52f466e27f0bc1d173194c7bea685dc241e6f112158481c68d633ba39a WatchSource:0}: Error finding container 73e9ff52f466e27f0bc1d173194c7bea685dc241e6f112158481c68d633ba39a: Status 404 returned error can't find the container with id 73e9ff52f466e27f0bc1d173194c7bea685dc241e6f112158481c68d633ba39a
Oct 30 21:22:56.078707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:56.078675    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj" event={"ID":"11f441d2-d449-4a47-9ec9-2af6280e89bb","Type":"ContainerStarted","Data":"73e9ff52f466e27f0bc1d173194c7bea685dc241e6f112158481c68d633ba39a"}
Oct 30 21:22:57.043651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.043613    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv" podUID="41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d" containerName="payload-processing" containerID="cri-o://03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c" gracePeriod=25
Oct 30 21:22:57.044119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.043613    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75" podUID="93a039b8-55d3-40eb-966a-b94255e14436" containerName="payload-processing" containerID="cri-o://3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9" gracePeriod=25
Oct 30 21:22:57.091355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.091309    4810 generic.go:356] "Generic (PLEG): container finished" podID="11f441d2-d449-4a47-9ec9-2af6280e89bb" containerID="3070172f53e0fb87457b8b719dd34d7ce42cd76d41df78d047d8da263486c450" exitCode=0
Oct 30 21:22:57.091548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.091529    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj" event={"ID":"11f441d2-d449-4a47-9ec9-2af6280e89bb","Type":"ContainerDied","Data":"3070172f53e0fb87457b8b719dd34d7ce42cd76d41df78d047d8da263486c450"}
Oct 30 21:22:57.386896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.386865    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:57.421228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.421201    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:57.442914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.442844    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-routing-overlay-volume\") pod \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") "
Oct 30 21:22:57.443568 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.443546    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-plugins-config-volume\") pod \"93a039b8-55d3-40eb-966a-b94255e14436\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") "
Oct 30 21:22:57.443980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.443958    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-kube-api-access-n9jh6\" (UniqueName: \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-kube-api-access-n9jh6\") pod \"93a039b8-55d3-40eb-966a-b94255e14436\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") "
Oct 30 21:22:57.444115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.443447    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-routing-overlay-volume" pod "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d" (UID: "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:57.444184 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.444084    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-plugins-config-volume" pod "93a039b8-55d3-40eb-966a-b94255e14436" (UID: "93a039b8-55d3-40eb-966a-b94255e14436"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:57.444313 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.444293    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials\") pod \"93a039b8-55d3-40eb-966a-b94255e14436\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") "
Oct 30 21:22:57.444531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.444503    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-plugins-config-volume\") pod \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") "
Oct 30 21:22:57.444643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.444595    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-kube-api-access-kn4hz\" (UniqueName: \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-kube-api-access-kn4hz\") pod \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") "
Oct 30 21:22:57.444643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.444636    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-routing-overlay-volume\") pod \"93a039b8-55d3-40eb-966a-b94255e14436\" (UID: \"93a039b8-55d3-40eb-966a-b94255e14436\") "
Oct 30 21:22:57.444941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.444686    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials\") pod \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\" (UID: \"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d\") "
Oct 30 21:22:57.445047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.444965    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-plugins-config-volume" pod "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d" (UID: "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:57.445328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.445306    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:57.445398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.445334    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:57.445398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.445352    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:57.445682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.445659    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-routing-overlay-volume" pod "93a039b8-55d3-40eb-966a-b94255e14436" (UID: "93a039b8-55d3-40eb-966a-b94255e14436"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:57.448026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.448003    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-kube-api-access-n9jh6" pod "93a039b8-55d3-40eb-966a-b94255e14436" (UID: "93a039b8-55d3-40eb-966a-b94255e14436"). InnerVolumeSpecName "kube-api-access-n9jh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:57.448094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.448039    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials" pod "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d" (UID: "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:57.448094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.448059    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials" pod "93a039b8-55d3-40eb-966a-b94255e14436" (UID: "93a039b8-55d3-40eb-966a-b94255e14436"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:57.448094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.448075    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-kube-api-access-kn4hz" pod "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d" (UID: "41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d"). InnerVolumeSpecName "kube-api-access-kn4hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:57.548594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.548553    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9jh6\" (UniqueName: \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-kube-api-access-n9jh6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:57.548594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.548593    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93a039b8-55d3-40eb-966a-b94255e14436-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:57.548914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.548610    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn4hz\" (UniqueName: \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-kube-api-access-kn4hz\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:57.548914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.548624    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93a039b8-55d3-40eb-966a-b94255e14436-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:57.548914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:57.548637    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.101693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.101591    4810 generic.go:356] "Generic (PLEG): container finished" podID="41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d" containerID="03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c" exitCode=0
Oct 30 21:22:58.101693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.101648    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"
Oct 30 21:22:58.101693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.101669    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv" event={"ID":"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d","Type":"ContainerDied","Data":"03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c"}
Oct 30 21:22:58.102334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.101724    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv" event={"ID":"41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d","Type":"ContainerDied","Data":"0ee8336c301157121b4a9097af979dc48755b61b440a75bfe329d358dfd8af61"}
Oct 30 21:22:58.102334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.101753    4810 scope.go:122] "RemoveContainer" containerID="03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c"
Oct 30 21:22:58.103629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.103606    4810 generic.go:356] "Generic (PLEG): container finished" podID="93a039b8-55d3-40eb-966a-b94255e14436" containerID="3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9" exitCode=0
Oct 30 21:22:58.103763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.103648    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"
Oct 30 21:22:58.103763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.103693    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75" event={"ID":"93a039b8-55d3-40eb-966a-b94255e14436","Type":"ContainerDied","Data":"3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9"}
Oct 30 21:22:58.103763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.103751    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75" event={"ID":"93a039b8-55d3-40eb-966a-b94255e14436","Type":"ContainerDied","Data":"9c3c0dbd23d99aa5b4372f591923f8a1c610d0f4107d03d37a89c5605a905fb9"}
Oct 30 21:22:58.117344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.117313    4810 scope.go:122] "RemoveContainer" containerID="03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c"
Oct 30 21:22:58.117681 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:58.117647    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c\": container with ID starting with 03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c not found: ID does not exist" containerID="03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c"
Oct 30 21:22:58.117794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.117680    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c"} err="failed to get container status \"03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c\": rpc error: code = NotFound desc = could not find container \"03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c\": container with ID starting with 03f9565dfd96563b9d4751779513e90f74ae41b16d129d5bddb369bf94ec829c not found: ID does not exist"
Oct 30 21:22:58.117794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.117698    4810 scope.go:122] "RemoveContainer" containerID="3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9"
Oct 30 21:22:58.127769 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.127735    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"]
Oct 30 21:22:58.132762 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.132738    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-85872tjsv"]
Oct 30 21:22:58.133806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.133790    4810 scope.go:122] "RemoveContainer" containerID="3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9"
Oct 30 21:22:58.134139 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:22:58.134120    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9\": container with ID starting with 3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9 not found: ID does not exist" containerID="3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9"
Oct 30 21:22:58.134193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.134145    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9"} err="failed to get container status \"3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9\": rpc error: code = NotFound desc = could not find container \"3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9\": container with ID starting with 3487d5b7f4a7b733392f49239cbd505aad29c1aad579ea25da700a154f2276b9 not found: ID does not exist"
Oct 30 21:22:58.142772 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.142743    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"]
Oct 30 21:22:58.148311 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.148285    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-6d58rmn75"]
Oct 30 21:22:58.275989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.275957    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:58.364271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.364198    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f441d2-d449-4a47-9ec9-2af6280e89bb-kube-api-access-wf5mp\" (UniqueName: \"kubernetes.io/projected/11f441d2-d449-4a47-9ec9-2af6280e89bb-kube-api-access-wf5mp\") pod \"11f441d2-d449-4a47-9ec9-2af6280e89bb\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") "
Oct 30 21:22:58.364271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.364259    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-xks-ca\") pod \"11f441d2-d449-4a47-9ec9-2af6280e89bb\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") "
Oct 30 21:22:58.364492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.364472    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-openshift-ca\") pod \"11f441d2-d449-4a47-9ec9-2af6280e89bb\" (UID: \"11f441d2-d449-4a47-9ec9-2af6280e89bb\") "
Oct 30 21:22:58.364759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.364735    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-openshift-ca" pod "11f441d2-d449-4a47-9ec9-2af6280e89bb" (UID: "11f441d2-d449-4a47-9ec9-2af6280e89bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:58.365073 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.365053    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.366422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.366397    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-xks-ca" pod "11f441d2-d449-4a47-9ec9-2af6280e89bb" (UID: "11f441d2-d449-4a47-9ec9-2af6280e89bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:58.366517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.366426    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f441d2-d449-4a47-9ec9-2af6280e89bb-kube-api-access-wf5mp" pod "11f441d2-d449-4a47-9ec9-2af6280e89bb" (UID: "11f441d2-d449-4a47-9ec9-2af6280e89bb"). InnerVolumeSpecName "kube-api-access-wf5mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:58.466701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.466665    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf5mp\" (UniqueName: \"kubernetes.io/projected/11f441d2-d449-4a47-9ec9-2af6280e89bb-kube-api-access-wf5mp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.466701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:58.466697    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f441d2-d449-4a47-9ec9-2af6280e89bb-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:59.128750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.128720    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"
Oct 30 21:22:59.129234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.128753    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj" event={"ID":"11f441d2-d449-4a47-9ec9-2af6280e89bb","Type":"ContainerDied","Data":"73e9ff52f466e27f0bc1d173194c7bea685dc241e6f112158481c68d633ba39a"}
Oct 30 21:22:59.129234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.128823    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73e9ff52f466e27f0bc1d173194c7bea685dc241e6f112158481c68d633ba39a"
Oct 30 21:22:59.664311 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.664275    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d" path="/var/lib/kubelet/pods/41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d/volumes"
Oct 30 21:22:59.664909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.664891    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a039b8-55d3-40eb-966a-b94255e14436" path="/var/lib/kubelet/pods/93a039b8-55d3-40eb-966a-b94255e14436/volumes"
Oct 30 21:22:59.681266 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.681240    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"]
Oct 30 21:22:59.682067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.682046    4810 cpu_manager.go:414] "containerMap: removing container" podUID="41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d" containerName="payload-processing"
Oct 30 21:22:59.682067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.682067    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d" containerName="payload-processing"
Oct 30 21:22:59.682242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.682101    4810 cpu_manager.go:414] "containerMap: removing container" podUID="93a039b8-55d3-40eb-966a-b94255e14436" containerName="payload-processing"
Oct 30 21:22:59.682242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.682109    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a039b8-55d3-40eb-966a-b94255e14436" containerName="payload-processing"
Oct 30 21:22:59.682242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.682145    4810 cpu_manager.go:414] "containerMap: removing container" podUID="11f441d2-d449-4a47-9ec9-2af6280e89bb" containerName="revoke-keys"
Oct 30 21:22:59.682242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.682152    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f441d2-d449-4a47-9ec9-2af6280e89bb" containerName="revoke-keys"
Oct 30 21:22:59.682448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.682396    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="41aa55ed-a43b-4ab7-9df5-7a60f5d10e0d" containerName="payload-processing"
Oct 30 21:22:59.682448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.682416    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a039b8-55d3-40eb-966a-b94255e14436" containerName="payload-processing"
Oct 30 21:22:59.682448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.682431    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f441d2-d449-4a47-9ec9-2af6280e89bb" containerName="revoke-keys"
Oct 30 21:22:59.687492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.687472    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:22:59.691647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.691621    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:59.691774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.691691    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:22:59.696296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.696274    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"]
Oct 30 21:22:59.788933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.788903    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7jtv\" (UniqueName: \"kubernetes.io/projected/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-kube-api-access-c7jtv\") pod \"maas-api-revoke-sub-159cec04-t29sq\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:22:59.789110 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.788960    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-159cec04-t29sq\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:22:59.789160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.789118    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-159cec04-t29sq\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:22:59.790390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.790341    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls" podUID="3a3257db-84f2-49b7-b136-73a5cc11e862" containerName="payload-processing" containerID="cri-o://27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082" gracePeriod=25
Oct 30 21:22:59.890791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.890756    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7jtv\" (UniqueName: \"kubernetes.io/projected/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-kube-api-access-c7jtv\") pod \"maas-api-revoke-sub-159cec04-t29sq\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:22:59.890965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.890822    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-159cec04-t29sq\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:22:59.890965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.890898    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-159cec04-t29sq\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:22:59.891569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.891545    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-159cec04-t29sq\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:22:59.893279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.893251    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-159cec04-t29sq\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:22:59.900105 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:22:59.900076    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7jtv\" (UniqueName: \"kubernetes.io/projected/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-kube-api-access-c7jtv\") pod \"maas-api-revoke-sub-159cec04-t29sq\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:23:00.008826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.008797    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:23:00.078950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.071923    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:23:00.144979 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.144944    4810 generic.go:356] "Generic (PLEG): container finished" podID="3a3257db-84f2-49b7-b136-73a5cc11e862" containerID="27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082" exitCode=0
Oct 30 21:23:00.145441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.145003    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"
Oct 30 21:23:00.145441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.145032    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls" event={"ID":"3a3257db-84f2-49b7-b136-73a5cc11e862","Type":"ContainerDied","Data":"27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082"}
Oct 30 21:23:00.145441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.145111    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls" event={"ID":"3a3257db-84f2-49b7-b136-73a5cc11e862","Type":"ContainerDied","Data":"5087e6ed87fde26caa940a051fc2dc8cbe42513039b10abb9fac0b42be8815cc"}
Oct 30 21:23:00.145441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.145159    4810 scope.go:122] "RemoveContainer" containerID="27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082"
Oct 30 21:23:00.155624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.155604    4810 scope.go:122] "RemoveContainer" containerID="27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082"
Oct 30 21:23:00.155937 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:00.155912    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082\": container with ID starting with 27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082 not found: ID does not exist" containerID="27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082"
Oct 30 21:23:00.156043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.155949    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082"} err="failed to get container status \"27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082\": rpc error: code = NotFound desc = could not find container \"27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082\": container with ID starting with 27162411234f2c9228ebdb8e7de8511e8530252553fb0cadff8ccb69644ce082 not found: ID does not exist"
Oct 30 21:23:00.188671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.188639    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk" podUID="066f80a8-e01a-465d-8dae-b6cfbaa4128a" containerName="payload-processing" containerID="cri-o://9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b" gracePeriod=25
Oct 30 21:23:00.199246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.199225    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-provider-credentials\") pod \"3a3257db-84f2-49b7-b136-73a5cc11e862\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") "
Oct 30 21:23:00.199343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.199326    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-plugins-config-volume\") pod \"3a3257db-84f2-49b7-b136-73a5cc11e862\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") "
Oct 30 21:23:00.199391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.199379    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-routing-overlay-volume\") pod \"3a3257db-84f2-49b7-b136-73a5cc11e862\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") "
Oct 30 21:23:00.199485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.199462    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-kube-api-access-gxpcg\" (UniqueName: \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-kube-api-access-gxpcg\") pod \"3a3257db-84f2-49b7-b136-73a5cc11e862\" (UID: \"3a3257db-84f2-49b7-b136-73a5cc11e862\") "
Oct 30 21:23:00.199804 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.199778    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-plugins-config-volume" pod "3a3257db-84f2-49b7-b136-73a5cc11e862" (UID: "3a3257db-84f2-49b7-b136-73a5cc11e862"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:00.200004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.199804    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-routing-overlay-volume" pod "3a3257db-84f2-49b7-b136-73a5cc11e862" (UID: "3a3257db-84f2-49b7-b136-73a5cc11e862"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:00.200226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.200198    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:00.200299 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.200227    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a3257db-84f2-49b7-b136-73a5cc11e862-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:00.201642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.201620    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-kube-api-access-gxpcg" pod "3a3257db-84f2-49b7-b136-73a5cc11e862" (UID: "3a3257db-84f2-49b7-b136-73a5cc11e862"). InnerVolumeSpecName "kube-api-access-gxpcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:00.201836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.201815    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-provider-credentials" pod "3a3257db-84f2-49b7-b136-73a5cc11e862" (UID: "3a3257db-84f2-49b7-b136-73a5cc11e862"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:00.336223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.330721    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:00.336223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.330761    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxpcg\" (UniqueName: \"kubernetes.io/projected/3a3257db-84f2-49b7-b136-73a5cc11e862-kube-api-access-gxpcg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:00.353908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.353878    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"]
Oct 30 21:23:00.481263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.481231    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"]
Oct 30 21:23:00.485655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.485625    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69d5m77ls"]
Oct 30 21:23:00.597189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.597162    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:23:00.737086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.737055    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-kube-api-access-lqw7j\" (UniqueName: \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-kube-api-access-lqw7j\") pod \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") "
Oct 30 21:23:00.737269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.737091    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-routing-overlay-volume\") pod \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") "
Oct 30 21:23:00.737269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.737117    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-provider-credentials\") pod \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") "
Oct 30 21:23:00.737269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.737187    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-plugins-config-volume\") pod \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\" (UID: \"066f80a8-e01a-465d-8dae-b6cfbaa4128a\") "
Oct 30 21:23:00.737603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.737572    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-routing-overlay-volume" pod "066f80a8-e01a-465d-8dae-b6cfbaa4128a" (UID: "066f80a8-e01a-465d-8dae-b6cfbaa4128a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:00.737603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.737592    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-plugins-config-volume" pod "066f80a8-e01a-465d-8dae-b6cfbaa4128a" (UID: "066f80a8-e01a-465d-8dae-b6cfbaa4128a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:00.739619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.739592    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-provider-credentials" pod "066f80a8-e01a-465d-8dae-b6cfbaa4128a" (UID: "066f80a8-e01a-465d-8dae-b6cfbaa4128a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:00.773136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.773109    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-kube-api-access-lqw7j" pod "066f80a8-e01a-465d-8dae-b6cfbaa4128a" (UID: "066f80a8-e01a-465d-8dae-b6cfbaa4128a"). InnerVolumeSpecName "kube-api-access-lqw7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:00.838846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.838794    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqw7j\" (UniqueName: \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-kube-api-access-lqw7j\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:00.838846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.838820    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:00.838846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.838830    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/066f80a8-e01a-465d-8dae-b6cfbaa4128a-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:00.838846 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:00.838838    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066f80a8-e01a-465d-8dae-b6cfbaa4128a-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:01.155227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.155197    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq" event={"ID":"c9b2f9f1-aeda-4825-8d00-74c511cce0c0","Type":"ContainerStarted","Data":"31c915baa0607fcaa22b6ea7d58f1b563b6505335aabd69910c3d0ca5bb9650d"}
Oct 30 21:23:01.155664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.155251    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq" event={"ID":"c9b2f9f1-aeda-4825-8d00-74c511cce0c0","Type":"ContainerStarted","Data":"44559637e3ba527124f44fce705ea1bd750db8f55f3be326b1099f41f751a2d3"}
Oct 30 21:23:01.156618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.156590    4810 generic.go:356] "Generic (PLEG): container finished" podID="066f80a8-e01a-465d-8dae-b6cfbaa4128a" containerID="9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b" exitCode=0
Oct 30 21:23:01.156715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.156635    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"
Oct 30 21:23:01.156715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.156656    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk" event={"ID":"066f80a8-e01a-465d-8dae-b6cfbaa4128a","Type":"ContainerDied","Data":"9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b"}
Oct 30 21:23:01.156715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.156702    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk" event={"ID":"066f80a8-e01a-465d-8dae-b6cfbaa4128a","Type":"ContainerDied","Data":"feebf4ed0cbecd46b194db84a83a15b58cf270dab03ed4be1adec694feed73c3"}
Oct 30 21:23:01.156830 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.156732    4810 scope.go:122] "RemoveContainer" containerID="9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b"
Oct 30 21:23:01.168158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.168139    4810 scope.go:122] "RemoveContainer" containerID="9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b"
Oct 30 21:23:01.168407 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:01.168386    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b\": container with ID starting with 9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b not found: ID does not exist" containerID="9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b"
Oct 30 21:23:01.168445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.168416    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b"} err="failed to get container status \"9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b\": rpc error: code = NotFound desc = could not find container \"9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b\": container with ID starting with 9e66f201761d1dcac847edb40b3ab163c16be1d40408d961f8e055493de1089b not found: ID does not exist"
Oct 30 21:23:01.175622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.175574    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq" podStartSLOduration=2.175557063 podStartE2EDuration="2.175557063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:01.172617326 +0000 UTC m=+2747.664433870" watchObservedRunningTime="2026-10-30 21:23:01.175557063 +0000 UTC m=+2747.667373609"
Oct 30 21:23:01.196153 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.196127    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"]
Oct 30 21:23:01.202347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.202323    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-56fcfplzk"]
Oct 30 21:23:01.270422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.270388    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"]
Oct 30 21:23:01.271188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.271167    4810 cpu_manager.go:414] "containerMap: removing container" podUID="066f80a8-e01a-465d-8dae-b6cfbaa4128a" containerName="payload-processing"
Oct 30 21:23:01.271188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.271183    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066f80a8-e01a-465d-8dae-b6cfbaa4128a" containerName="payload-processing"
Oct 30 21:23:01.271362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.271229    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3a3257db-84f2-49b7-b136-73a5cc11e862" containerName="payload-processing"
Oct 30 21:23:01.271362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.271235    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3257db-84f2-49b7-b136-73a5cc11e862" containerName="payload-processing"
Oct 30 21:23:01.271438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.271364    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3257db-84f2-49b7-b136-73a5cc11e862" containerName="payload-processing"
Oct 30 21:23:01.271438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.271379    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="066f80a8-e01a-465d-8dae-b6cfbaa4128a" containerName="payload-processing"
Oct 30 21:23:01.281923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.277062    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.301596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.301552    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"]
Oct 30 21:23:01.450214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.450134    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lff\" (UniqueName: \"kubernetes.io/projected/94d23be3-ef68-49b7-aa8b-6390aa3afec8-kube-api-access-p8lff\") pod \"maas-api-revoke-sub-3f29c044-77c2r\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.450351 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.450239    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f29c044-77c2r\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.450351 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.450275    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f29c044-77c2r\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.551303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.551267    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lff\" (UniqueName: \"kubernetes.io/projected/94d23be3-ef68-49b7-aa8b-6390aa3afec8-kube-api-access-p8lff\") pod \"maas-api-revoke-sub-3f29c044-77c2r\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.551461 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.551346    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f29c044-77c2r\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.551461 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.551369    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f29c044-77c2r\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.552039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.552011    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f29c044-77c2r\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.553732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.553710    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f29c044-77c2r\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.560328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.560303    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lff\" (UniqueName: \"kubernetes.io/projected/94d23be3-ef68-49b7-aa8b-6390aa3afec8-kube-api-access-p8lff\") pod \"maas-api-revoke-sub-3f29c044-77c2r\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.603249 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.603210    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:01.685186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.675356    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066f80a8-e01a-465d-8dae-b6cfbaa4128a" path="/var/lib/kubelet/pods/066f80a8-e01a-465d-8dae-b6cfbaa4128a/volumes"
Oct 30 21:23:01.685186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.676015    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3257db-84f2-49b7-b136-73a5cc11e862" path="/var/lib/kubelet/pods/3a3257db-84f2-49b7-b136-73a5cc11e862/volumes"
Oct 30 21:23:01.894395 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:01.894361    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d23be3_ef68_49b7_aa8b_6390aa3afec8.slice/crio-352317030e59046d02f65b358fcbdfb25ac1d1edd4a9a60fe5f4f59228500b33 WatchSource:0}: Error finding container 352317030e59046d02f65b358fcbdfb25ac1d1edd4a9a60fe5f4f59228500b33: Status 404 returned error can't find the container with id 352317030e59046d02f65b358fcbdfb25ac1d1edd4a9a60fe5f4f59228500b33
Oct 30 21:23:01.895480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:01.895448    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"]
Oct 30 21:23:02.167165 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:02.167079    4810 generic.go:356] "Generic (PLEG): container finished" podID="c9b2f9f1-aeda-4825-8d00-74c511cce0c0" containerID="31c915baa0607fcaa22b6ea7d58f1b563b6505335aabd69910c3d0ca5bb9650d" exitCode=0
Oct 30 21:23:02.167600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:02.167166    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq" event={"ID":"c9b2f9f1-aeda-4825-8d00-74c511cce0c0","Type":"ContainerDied","Data":"31c915baa0607fcaa22b6ea7d58f1b563b6505335aabd69910c3d0ca5bb9650d"}
Oct 30 21:23:02.169539 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:02.169516    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r" event={"ID":"94d23be3-ef68-49b7-aa8b-6390aa3afec8","Type":"ContainerStarted","Data":"352317030e59046d02f65b358fcbdfb25ac1d1edd4a9a60fe5f4f59228500b33"}
Oct 30 21:23:03.179665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.179626    4810 generic.go:356] "Generic (PLEG): container finished" podID="94d23be3-ef68-49b7-aa8b-6390aa3afec8" containerID="898039dfe63c3e643237e1b02d1a10e1bfd381845f5e7d7766053500e5d11901" exitCode=0
Oct 30 21:23:03.180106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.179709    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r" event={"ID":"94d23be3-ef68-49b7-aa8b-6390aa3afec8","Type":"ContainerDied","Data":"898039dfe63c3e643237e1b02d1a10e1bfd381845f5e7d7766053500e5d11901"}
Oct 30 21:23:03.505881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.504234    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:23:03.603437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.603394    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-openshift-ca\") pod \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") "
Oct 30 21:23:03.603644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.603487    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-kube-api-access-c7jtv\" (UniqueName: \"kubernetes.io/projected/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-kube-api-access-c7jtv\") pod \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") "
Oct 30 21:23:03.603699 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.603650    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-xks-ca\") pod \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\" (UID: \"c9b2f9f1-aeda-4825-8d00-74c511cce0c0\") "
Oct 30 21:23:03.604817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.604795    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-openshift-ca" pod "c9b2f9f1-aeda-4825-8d00-74c511cce0c0" (UID: "c9b2f9f1-aeda-4825-8d00-74c511cce0c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:03.606348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.606320    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-xks-ca" pod "c9b2f9f1-aeda-4825-8d00-74c511cce0c0" (UID: "c9b2f9f1-aeda-4825-8d00-74c511cce0c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:03.607637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.607598    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-kube-api-access-c7jtv" pod "c9b2f9f1-aeda-4825-8d00-74c511cce0c0" (UID: "c9b2f9f1-aeda-4825-8d00-74c511cce0c0"). InnerVolumeSpecName "kube-api-access-c7jtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:03.705590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.705514    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:03.705590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.705545    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:03.705590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:03.705557    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7jtv\" (UniqueName: \"kubernetes.io/projected/c9b2f9f1-aeda-4825-8d00-74c511cce0c0-kube-api-access-c7jtv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.188970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.188933    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq" event={"ID":"c9b2f9f1-aeda-4825-8d00-74c511cce0c0","Type":"ContainerDied","Data":"44559637e3ba527124f44fce705ea1bd750db8f55f3be326b1099f41f751a2d3"}
Oct 30 21:23:04.188970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.188955    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"
Oct 30 21:23:04.189416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.188998    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44559637e3ba527124f44fce705ea1bd750db8f55f3be326b1099f41f751a2d3"
Oct 30 21:23:04.353211 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.353184    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:04.518524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.518439    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d23be3-ef68-49b7-aa8b-6390aa3afec8-kube-api-access-p8lff\" (UniqueName: \"kubernetes.io/projected/94d23be3-ef68-49b7-aa8b-6390aa3afec8-kube-api-access-p8lff\") pod \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") "
Oct 30 21:23:04.518524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.518494    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-xks-ca\") pod \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") "
Oct 30 21:23:04.518773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.518533    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-openshift-ca\") pod \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\" (UID: \"94d23be3-ef68-49b7-aa8b-6390aa3afec8\") "
Oct 30 21:23:04.519027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.518999    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-openshift-ca" pod "94d23be3-ef68-49b7-aa8b-6390aa3afec8" (UID: "94d23be3-ef68-49b7-aa8b-6390aa3afec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:04.519189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.519171    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.521284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.521261    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-xks-ca" pod "94d23be3-ef68-49b7-aa8b-6390aa3afec8" (UID: "94d23be3-ef68-49b7-aa8b-6390aa3afec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:04.521403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.521298    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d23be3-ef68-49b7-aa8b-6390aa3afec8-kube-api-access-p8lff" pod "94d23be3-ef68-49b7-aa8b-6390aa3afec8" (UID: "94d23be3-ef68-49b7-aa8b-6390aa3afec8"). InnerVolumeSpecName "kube-api-access-p8lff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:04.620133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.620103    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8lff\" (UniqueName: \"kubernetes.io/projected/94d23be3-ef68-49b7-aa8b-6390aa3afec8-kube-api-access-p8lff\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.620133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.620130    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d23be3-ef68-49b7-aa8b-6390aa3afec8-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.665391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.665361    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"]
Oct 30 21:23:04.666238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.666215    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c9b2f9f1-aeda-4825-8d00-74c511cce0c0" containerName="revoke-keys"
Oct 30 21:23:04.666238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.666236    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b2f9f1-aeda-4825-8d00-74c511cce0c0" containerName="revoke-keys"
Oct 30 21:23:04.666436 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.666251    4810 cpu_manager.go:414] "containerMap: removing container" podUID="94d23be3-ef68-49b7-aa8b-6390aa3afec8" containerName="revoke-keys"
Oct 30 21:23:04.666436 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.666258    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d23be3-ef68-49b7-aa8b-6390aa3afec8" containerName="revoke-keys"
Oct 30 21:23:04.666552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.666516    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b2f9f1-aeda-4825-8d00-74c511cce0c0" containerName="revoke-keys"
Oct 30 21:23:04.666552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.666532    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d23be3-ef68-49b7-aa8b-6390aa3afec8" containerName="revoke-keys"
Oct 30 21:23:04.670563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.670538    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:04.680064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.680035    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"]
Oct 30 21:23:04.827233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.827132    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74ced40b-2glv9\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:04.827233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.827203    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4c7\" (UniqueName: \"kubernetes.io/projected/55c38e4a-efb4-4d68-871e-a509a984d293-kube-api-access-jb4c7\") pod \"maas-api-revoke-sub-74ced40b-2glv9\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:04.827466 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.827266    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74ced40b-2glv9\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:04.929396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.929349    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74ced40b-2glv9\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:04.929604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.929414    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4c7\" (UniqueName: \"kubernetes.io/projected/55c38e4a-efb4-4d68-871e-a509a984d293-kube-api-access-jb4c7\") pod \"maas-api-revoke-sub-74ced40b-2glv9\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:04.929604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.929453    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74ced40b-2glv9\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:04.930085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.930060    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74ced40b-2glv9\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:04.931936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.931915    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74ced40b-2glv9\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:04.937945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.937917    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4c7\" (UniqueName: \"kubernetes.io/projected/55c38e4a-efb4-4d68-871e-a509a984d293-kube-api-access-jb4c7\") pod \"maas-api-revoke-sub-74ced40b-2glv9\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:04.989541 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:04.989513    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:05.212646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:05.212613    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r" event={"ID":"94d23be3-ef68-49b7-aa8b-6390aa3afec8","Type":"ContainerDied","Data":"352317030e59046d02f65b358fcbdfb25ac1d1edd4a9a60fe5f4f59228500b33"}
Oct 30 21:23:05.213133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:05.213111    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="352317030e59046d02f65b358fcbdfb25ac1d1edd4a9a60fe5f4f59228500b33"
Oct 30 21:23:05.213253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:05.213199    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"
Oct 30 21:23:05.275533 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:05.275503    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c38e4a_efb4_4d68_871e_a509a984d293.slice/crio-55934b15ff40313e71cf18ece1e14948ddde455aae0480a4dc97a971f9576c1e WatchSource:0}: Error finding container 55934b15ff40313e71cf18ece1e14948ddde455aae0480a4dc97a971f9576c1e: Status 404 returned error can't find the container with id 55934b15ff40313e71cf18ece1e14948ddde455aae0480a4dc97a971f9576c1e
Oct 30 21:23:05.276532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:05.276453    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"]
Oct 30 21:23:05.776402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:05.776368    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"]
Oct 30 21:23:05.781015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:05.780995    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:05.790687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:05.790658    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"]
Oct 30 21:23:05.942598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:05.942561    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa51bea-8zv4t\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:05.942766 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:05.942744    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8crg\" (UniqueName: \"kubernetes.io/projected/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-kube-api-access-w8crg\") pod \"maas-api-revoke-sub-1fa51bea-8zv4t\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:05.942832 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:05.942800    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa51bea-8zv4t\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:06.044530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.044498    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa51bea-8zv4t\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:06.044729 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.044649    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa51bea-8zv4t\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:06.044834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.044817    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8crg\" (UniqueName: \"kubernetes.io/projected/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-kube-api-access-w8crg\") pod \"maas-api-revoke-sub-1fa51bea-8zv4t\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:06.045405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.045380    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa51bea-8zv4t\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:06.047350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.047324    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa51bea-8zv4t\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:06.053452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.053427    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8crg\" (UniqueName: \"kubernetes.io/projected/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-kube-api-access-w8crg\") pod \"maas-api-revoke-sub-1fa51bea-8zv4t\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:06.100744 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.100716    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:06.228815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.228719    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9" event={"ID":"55c38e4a-efb4-4d68-871e-a509a984d293","Type":"ContainerStarted","Data":"e348b20e3e83231a0263154de6a4b1d569c6bd6a40c5cc3c7f76e507f207e77b"}
Oct 30 21:23:06.228815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.228783    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9" event={"ID":"55c38e4a-efb4-4d68-871e-a509a984d293","Type":"ContainerStarted","Data":"55934b15ff40313e71cf18ece1e14948ddde455aae0480a4dc97a971f9576c1e"}
Oct 30 21:23:06.245204 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.245157    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9" podStartSLOduration=2.245143317 podStartE2EDuration="2.245143317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:06.24424405 +0000 UTC m=+2752.736060599" watchObservedRunningTime="2026-10-30 21:23:06.245143317 +0000 UTC m=+2752.736959891"
Oct 30 21:23:06.382569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:06.381851    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"]
Oct 30 21:23:06.383685 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:06.383644    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155a6d44_f3a1_4c4a_9f89_075655b8cb4c.slice/crio-6c00961554daf29933d4b1ecb4c4ce2556b323db25a6588e995c6fe22220e504 WatchSource:0}: Error finding container 6c00961554daf29933d4b1ecb4c4ce2556b323db25a6588e995c6fe22220e504: Status 404 returned error can't find the container with id 6c00961554daf29933d4b1ecb4c4ce2556b323db25a6588e995c6fe22220e504
Oct 30 21:23:07.238616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:07.238523    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t" event={"ID":"155a6d44-f3a1-4c4a-9f89-075655b8cb4c","Type":"ContainerStarted","Data":"e4b7e31026d4e5af47af0ac7e977f4d8ee6f2d742f049fd799ae60638d42e82d"}
Oct 30 21:23:07.239100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:07.238624    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t" event={"ID":"155a6d44-f3a1-4c4a-9f89-075655b8cb4c","Type":"ContainerStarted","Data":"6c00961554daf29933d4b1ecb4c4ce2556b323db25a6588e995c6fe22220e504"}
Oct 30 21:23:07.240500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:07.240477    4810 generic.go:356] "Generic (PLEG): container finished" podID="55c38e4a-efb4-4d68-871e-a509a984d293" containerID="e348b20e3e83231a0263154de6a4b1d569c6bd6a40c5cc3c7f76e507f207e77b" exitCode=0
Oct 30 21:23:07.240631 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:07.240558    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9" event={"ID":"55c38e4a-efb4-4d68-871e-a509a984d293","Type":"ContainerDied","Data":"e348b20e3e83231a0263154de6a4b1d569c6bd6a40c5cc3c7f76e507f207e77b"}
Oct 30 21:23:07.258775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:07.258730    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t" podStartSLOduration=2.258716902 podStartE2EDuration="2.258716902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:07.255756132 +0000 UTC m=+2753.747572678" watchObservedRunningTime="2026-10-30 21:23:07.258716902 +0000 UTC m=+2753.750533446"
Oct 30 21:23:08.253905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.253852    4810 generic.go:356] "Generic (PLEG): container finished" podID="155a6d44-f3a1-4c4a-9f89-075655b8cb4c" containerID="e4b7e31026d4e5af47af0ac7e977f4d8ee6f2d742f049fd799ae60638d42e82d" exitCode=0
Oct 30 21:23:08.254396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.253937    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t" event={"ID":"155a6d44-f3a1-4c4a-9f89-075655b8cb4c","Type":"ContainerDied","Data":"e4b7e31026d4e5af47af0ac7e977f4d8ee6f2d742f049fd799ae60638d42e82d"}
Oct 30 21:23:08.509022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.508957    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:08.578689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.578653    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-openshift-ca\") pod \"55c38e4a-efb4-4d68-871e-a509a984d293\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") "
Oct 30 21:23:08.578878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.578803    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c38e4a-efb4-4d68-871e-a509a984d293-kube-api-access-jb4c7\" (UniqueName: \"kubernetes.io/projected/55c38e4a-efb4-4d68-871e-a509a984d293-kube-api-access-jb4c7\") pod \"55c38e4a-efb4-4d68-871e-a509a984d293\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") "
Oct 30 21:23:08.578968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.578951    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-xks-ca\") pod \"55c38e4a-efb4-4d68-871e-a509a984d293\" (UID: \"55c38e4a-efb4-4d68-871e-a509a984d293\") "
Oct 30 21:23:08.579107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.579082    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-openshift-ca" pod "55c38e4a-efb4-4d68-871e-a509a984d293" (UID: "55c38e4a-efb4-4d68-871e-a509a984d293"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:08.579640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.579617    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:08.581045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.581014    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c38e4a-efb4-4d68-871e-a509a984d293-kube-api-access-jb4c7" pod "55c38e4a-efb4-4d68-871e-a509a984d293" (UID: "55c38e4a-efb4-4d68-871e-a509a984d293"). InnerVolumeSpecName "kube-api-access-jb4c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:08.581135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.581085    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-xks-ca" pod "55c38e4a-efb4-4d68-871e-a509a984d293" (UID: "55c38e4a-efb4-4d68-871e-a509a984d293"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:08.681362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.681334    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb4c7\" (UniqueName: \"kubernetes.io/projected/55c38e4a-efb4-4d68-871e-a509a984d293-kube-api-access-jb4c7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:08.681362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:08.681362    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c38e4a-efb4-4d68-871e-a509a984d293-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:09.264129 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.264097    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"
Oct 30 21:23:09.264569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.264127    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9" event={"ID":"55c38e4a-efb4-4d68-871e-a509a984d293","Type":"ContainerDied","Data":"55934b15ff40313e71cf18ece1e14948ddde455aae0480a4dc97a971f9576c1e"}
Oct 30 21:23:09.264569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.264187    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55934b15ff40313e71cf18ece1e14948ddde455aae0480a4dc97a971f9576c1e"
Oct 30 21:23:09.435778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.435748    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:09.493674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.493641    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-xks-ca\") pod \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") "
Oct 30 21:23:09.493998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.493976    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-openshift-ca\") pod \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") "
Oct 30 21:23:09.494163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.494146    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-kube-api-access-w8crg\" (UniqueName: \"kubernetes.io/projected/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-kube-api-access-w8crg\") pod \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\" (UID: \"155a6d44-f3a1-4c4a-9f89-075655b8cb4c\") "
Oct 30 21:23:09.494742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.494712    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-openshift-ca" pod "155a6d44-f3a1-4c4a-9f89-075655b8cb4c" (UID: "155a6d44-f3a1-4c4a-9f89-075655b8cb4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:09.495063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.495041    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:09.497039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.497012    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-xks-ca" pod "155a6d44-f3a1-4c4a-9f89-075655b8cb4c" (UID: "155a6d44-f3a1-4c4a-9f89-075655b8cb4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:09.497450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.497427    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-kube-api-access-w8crg" pod "155a6d44-f3a1-4c4a-9f89-075655b8cb4c" (UID: "155a6d44-f3a1-4c4a-9f89-075655b8cb4c"). InnerVolumeSpecName "kube-api-access-w8crg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:09.596726 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.596648    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:09.596726 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:09.596679    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8crg\" (UniqueName: \"kubernetes.io/projected/155a6d44-f3a1-4c4a-9f89-075655b8cb4c-kube-api-access-w8crg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:10.273721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.273682    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t" event={"ID":"155a6d44-f3a1-4c4a-9f89-075655b8cb4c","Type":"ContainerDied","Data":"6c00961554daf29933d4b1ecb4c4ce2556b323db25a6588e995c6fe22220e504"}
Oct 30 21:23:10.273721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.273706    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"
Oct 30 21:23:10.274317 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.273745    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c00961554daf29933d4b1ecb4c4ce2556b323db25a6588e995c6fe22220e504"
Oct 30 21:23:10.858385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.846534    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"]
Oct 30 21:23:10.858385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.847731    4810 cpu_manager.go:414] "containerMap: removing container" podUID="55c38e4a-efb4-4d68-871e-a509a984d293" containerName="revoke-keys"
Oct 30 21:23:10.858385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.847764    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c38e4a-efb4-4d68-871e-a509a984d293" containerName="revoke-keys"
Oct 30 21:23:10.858385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.847843    4810 cpu_manager.go:414] "containerMap: removing container" podUID="155a6d44-f3a1-4c4a-9f89-075655b8cb4c" containerName="revoke-keys"
Oct 30 21:23:10.858385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.847851    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155a6d44-f3a1-4c4a-9f89-075655b8cb4c" containerName="revoke-keys"
Oct 30 21:23:10.858385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.848065    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="155a6d44-f3a1-4c4a-9f89-075655b8cb4c" containerName="revoke-keys"
Oct 30 21:23:10.858385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.848089    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c38e4a-efb4-4d68-871e-a509a984d293" containerName="revoke-keys"
Oct 30 21:23:10.858385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.852961    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:10.916960 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.916924    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"]
Oct 30 21:23:10.949175 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.943090    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-routing-overlay-volume\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:10.949491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.949460    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qzkj\" (UniqueName: \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-kube-api-access-5qzkj\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:10.949670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.949653    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:10.949813 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:10.949798    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:11.053442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:11.051948    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:11.053442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:11.052235    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-routing-overlay-volume\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:11.053442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:11.052300    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qzkj\" (UniqueName: \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-kube-api-access-5qzkj\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:11.053442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:11.052359    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:11.054233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:11.054205    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-routing-overlay-volume\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:11.054373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:11.054339    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:11.055309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:11.055282    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:11.061348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:11.061328    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qzkj\" (UniqueName: \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-kube-api-access-5qzkj\") pod \"payload-processing-external-model-ff5d646d-d5xlp\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") " pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:11.216500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:11.216458    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:11.504191 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:11.504160    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"]
Oct 30 21:23:11.505001 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:11.504975    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b6cffb_4d93_40d9_9a81_9ee452617378.slice/crio-abb61ec2c83376159e230f1e5f7fc1b24cd0fbe7213ce9b31cc3685156608aba WatchSource:0}: Error finding container abb61ec2c83376159e230f1e5f7fc1b24cd0fbe7213ce9b31cc3685156608aba: Status 404 returned error can't find the container with id abb61ec2c83376159e230f1e5f7fc1b24cd0fbe7213ce9b31cc3685156608aba
Oct 30 21:23:12.309416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:12.309376    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"]
Oct 30 21:23:12.324277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:12.323193    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"]
Oct 30 21:23:12.395576 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:12.395538    4810 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp" secret="" err="secret \"payload-processing-external-model-dockercfg-sfdtt\" not found"
Oct 30 21:23:12.395962 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:12.395931    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp" event={"ID":"a5b6cffb-4d93-40d9-9a81-9ee452617378","Type":"ContainerStarted","Data":"7fa6a40f18ac96cce4dcd0aaf190e31f28690c18f03da80b601288e53cec3ecf"}
Oct 30 21:23:12.396073 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:12.395996    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp" event={"ID":"a5b6cffb-4d93-40d9-9a81-9ee452617378","Type":"ContainerStarted","Data":"abb61ec2c83376159e230f1e5f7fc1b24cd0fbe7213ce9b31cc3685156608aba"}
Oct 30 21:23:12.396073 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:12.396021    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:12.415635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:12.415579    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp" podStartSLOduration=2.415561543 podStartE2EDuration="2.415561543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:12.411241994 +0000 UTC m=+2758.903058539" watchObservedRunningTime="2026-10-30 21:23:12.415561543 +0000 UTC m=+2758.907378089"
Oct 30 21:23:12.471351 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:12.471317    4810 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Oct 30 21:23:12.471507 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:12.471401    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume podName:a5b6cffb-4d93-40d9-9a81-9ee452617378 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:12.971379991 +0000 UTC m=+2759.463196532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume") pod "payload-processing-external-model-ff5d646d-d5xlp" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378") : configmap "payload-processing-external-model-plugins" not found
Oct 30 21:23:12.471507 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:12.471333    4810 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b4eff0c43a87: secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:12.471507 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:12.471439    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp: secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:12.471507 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:12.471494    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials podName:a5b6cffb-4d93-40d9-9a81-9ee452617378 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:12.97147739 +0000 UTC m=+2759.463293930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials") pod "payload-processing-external-model-ff5d646d-d5xlp" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378") : secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:12.986248 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:12.985747    4810 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b4eff0c43a87: secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:12.986248 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:12.985778    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp: secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:12.986248 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:12.985797    4810 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Oct 30 21:23:12.986248 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:12.985843    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials podName:a5b6cffb-4d93-40d9-9a81-9ee452617378 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:13.985824178 +0000 UTC m=+2760.477640706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials") pod "payload-processing-external-model-ff5d646d-d5xlp" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378") : secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:12.986248 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:12.985877    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume podName:a5b6cffb-4d93-40d9-9a81-9ee452617378 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:13.985852319 +0000 UTC m=+2760.477668850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume") pod "payload-processing-external-model-ff5d646d-d5xlp" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378") : configmap "payload-processing-external-model-plugins" not found
Oct 30 21:23:13.228972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.228827    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"]
Oct 30 21:23:13.235053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.235025    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.253274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.246795    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ddcf93-openshift-default-dockercfg-qj8w6\""
Oct 30 21:23:13.254852 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.254807    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"]
Oct 30 21:23:13.294961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.291931    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-credential-socket\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.294961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.291988    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmwjx\" (UniqueName: \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-kube-api-access-vmwjx\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.294961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.292019    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-certs\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.294961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.292069    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-podinfo\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.294961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.292215    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-data\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.294961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.292246    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-socket\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.294961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.292325    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istiod-ca-cert\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.294961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.292389    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-envoy\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.294961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.292420    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-token\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.377957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.377922    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"]
Oct 30 21:23:13.405881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.401922    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-data\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.405881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.401984    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-socket\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.405881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.402071    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istiod-ca-cert\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.405881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.402153    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-envoy\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.405881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.402189    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-token\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.405881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.402274    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-credential-socket\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.405881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.402318    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmwjx\" (UniqueName: \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-kube-api-access-vmwjx\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.405881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.402354    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-certs\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.405881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.402418    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-podinfo\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.417896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.407144    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.417896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.412253    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-data\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.440362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.440322    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-socket\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.447122 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.442200    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-credential-socket\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.450633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.443255    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istiod-ca-cert\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.454452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.443776    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-certs\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.508927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.505675    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmwjx\" (UniqueName: \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-kube-api-access-vmwjx\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.508927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.506716    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69xp7\" (UniqueName: \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-kube-api-access-69xp7\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.508927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.506790    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.508927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.506872    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.508927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.507144    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.554481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.544558    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"]
Oct 30 21:23:13.565950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.545176    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-podinfo\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.566163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.547821    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-token\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.566255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.555020    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-envoy\") pod \"e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") " pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.600984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.600949    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:13.636357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.636314    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.636579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.636559    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69xp7\" (UniqueName: \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-kube-api-access-69xp7\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.636668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.636623    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.636731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.636681    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.638623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.638589    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.638806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.638778    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.649802 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.648403    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.649802 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.649465    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69xp7\" (UniqueName: \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-kube-api-access-69xp7\") pod \"payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") " pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.920021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.917528    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:13.939845 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:13.939811    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9fbac4_7de9_488e_8181_6b9be2b154dd.slice/crio-bfdef695d829dbdc2f199b680f220761b4595046e367b9a6668e1be3d9d1d2cd WatchSource:0}: Error finding container bfdef695d829dbdc2f199b680f220761b4595046e367b9a6668e1be3d9d1d2cd: Status 404 returned error can't find the container with id bfdef695d829dbdc2f199b680f220761b4595046e367b9a6668e1be3d9d1d2cd
Oct 30 21:23:13.940622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.940592    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"]
Oct 30 21:23:13.943158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.943118    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:13.943261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.943200    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:13.943261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:13.943243    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:14.069082 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:14.069038    4810 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b4eff0c43a87: secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:14.069082 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:14.069083    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp: secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:14.069445 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:14.069126    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials podName:a5b6cffb-4d93-40d9-9a81-9ee452617378 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:16.069112439 +0000 UTC m=+2762.560928975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials") pod "payload-processing-external-model-ff5d646d-d5xlp" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378") : secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:14.069445 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:14.069198    4810 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Oct 30 21:23:14.069445 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:14.069281    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume podName:a5b6cffb-4d93-40d9-9a81-9ee452617378 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:16.069261332 +0000 UTC m=+2762.561077878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume") pod "payload-processing-external-model-ff5d646d-d5xlp" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378") : configmap "payload-processing-external-model-plugins" not found
Oct 30 21:23:14.185605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.185510    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"]
Oct 30 21:23:14.195458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.195432    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.207303 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.205364    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"]
Oct 30 21:23:14.246992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.246962    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"]
Oct 30 21:23:14.247392 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:14.247359    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f86b98_ce6e_42a5_9177_740f98ab1b17.slice/crio-b4546caa4c3d63a8270a1df6db13c5683ace67ffc730ee1e4a30c1723879b445 WatchSource:0}: Error finding container b4546caa4c3d63a8270a1df6db13c5683ace67ffc730ee1e4a30c1723879b445: Status 404 returned error can't find the container with id b4546caa4c3d63a8270a1df6db13c5683ace67ffc730ee1e4a30c1723879b445
Oct 30 21:23:14.278913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.278313    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.278913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.278401    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn489\" (UniqueName: \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-kube-api-access-mn489\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.278913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.278586    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.278913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.278708    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.380689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.380651    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.381066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.381033    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.381268 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.381253    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn489\" (UniqueName: \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-kube-api-access-mn489\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.381595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.381567    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.382270 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.382244    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.382562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.382543    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.384075 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.384050    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.390222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.390196    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn489\" (UniqueName: \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-kube-api-access-mn489\") pod \"payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") " pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.547639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.547571    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:14.642906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.623467    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" event={"ID":"ec9fbac4-7de9-488e-8181-6b9be2b154dd","Type":"ContainerStarted","Data":"378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732"}
Oct 30 21:23:14.642906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.623546    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" event={"ID":"ec9fbac4-7de9-488e-8181-6b9be2b154dd","Type":"ContainerStarted","Data":"bfdef695d829dbdc2f199b680f220761b4595046e367b9a6668e1be3d9d1d2cd"}
Oct 30 21:23:14.648841 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.647030    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" event={"ID":"45f86b98-ce6e-42a5-9177-740f98ab1b17","Type":"ContainerStarted","Data":"c7ae0f27821e8f02a8c1f5f5fea95d6bbbf5e90068cc7cd07f27dfab61bb07b0"}
Oct 30 21:23:14.648841 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.647099    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" event={"ID":"45f86b98-ce6e-42a5-9177-740f98ab1b17","Type":"ContainerStarted","Data":"b4546caa4c3d63a8270a1df6db13c5683ace67ffc730ee1e4a30c1723879b445"}
Oct 30 21:23:14.648841 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.647943    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:14.675904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.674151    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" podStartSLOduration=1.674131445 podStartE2EDuration="1.674131445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:14.664414196 +0000 UTC m=+2761.156230742" watchObservedRunningTime="2026-10-30 21:23:14.674131445 +0000 UTC m=+2761.165947992"
Oct 30 21:23:14.687551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.687497    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" podStartSLOduration=1.687477585 podStartE2EDuration="1.687477585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:14.680776735 +0000 UTC m=+2761.172593281" watchObservedRunningTime="2026-10-30 21:23:14.687477585 +0000 UTC m=+2761.179294131"
Oct 30 21:23:14.703993 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:14.703530    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-aaf84e248242: secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:14.703993 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:14.703555    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z: secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:14.703993 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:14.703606    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials podName:45f86b98-ce6e-42a5-9177-740f98ab1b17 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:15.203588204 +0000 UTC m=+2761.695404741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17") : secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:14.899851 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.899173    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"]
Oct 30 21:23:14.907044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.906965    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"]
Oct 30 21:23:14.915018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:14.911542    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"]
Oct 30 21:23:14.915622 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:14.915597    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e4ded2_6260_4cb9_942f_99e66504e118.slice/crio-6ac023dfe34aafd07cc38891f0c9c4d7a63cee491d3b6965cc82b3e1685dee99 WatchSource:0}: Error finding container 6ac023dfe34aafd07cc38891f0c9c4d7a63cee491d3b6965cc82b3e1685dee99: Status 404 returned error can't find the container with id 6ac023dfe34aafd07cc38891f0c9c4d7a63cee491d3b6965cc82b3e1685dee99
Oct 30 21:23:15.222911 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:15.219283    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-aaf84e248242: secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:15.222911 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:15.219317    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z: secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:15.222911 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:15.219379    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials podName:45f86b98-ce6e-42a5-9177-740f98ab1b17 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:16.219361077 +0000 UTC m=+2762.711177620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17") : secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:15.497477 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.497395    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"]
Oct 30 21:23:15.508187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.508157    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.509692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.509658    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"]
Oct 30 21:23:15.512722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.512364    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-564ea4943d1a\""
Oct 30 21:23:15.512722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.512376    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zj7ss\""
Oct 30 21:23:15.527185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.527142    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-provider-credentials\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.527354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.527335    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp525\" (UniqueName: \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-kube-api-access-vp525\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.527443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.527428    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-plugins-config-volume\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.527586 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.527570    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-routing-overlay-volume\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.576877 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.569951    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"]
Oct 30 21:23:15.584543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.584515    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"]
Oct 30 21:23:15.607703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.607673    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:15.616176 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.616145    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:15.628810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.628781    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp525\" (UniqueName: \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-kube-api-access-vp525\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.628981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.628920    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-plugins-config-volume\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.629068 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.629051    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-routing-overlay-volume\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.629193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.629175    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-provider-credentials\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.630659 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.630637    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-plugins-config-volume\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.631175 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.631156    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-routing-overlay-volume\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.635708 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.635632    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-provider-credentials\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.639898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.639840    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp525\" (UniqueName: \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-kube-api-access-vp525\") pod \"payload-processing-external-model-5757ff5f4d-kj29g\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:15.662294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.662268    4810 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" secret="" err="secret \"payload-processing-external-model-e2e--789fdc40-dockercfg-5ll48\" not found"
Oct 30 21:23:15.669022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.669001    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:15.669133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.669043    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr" event={"ID":"a7e4ded2-6260-4cb9-942f-99e66504e118","Type":"ContainerStarted","Data":"a305a2ae512f7d008a57afdb56fdc0576f65c2188ce72cb785a6e3e0d0f6e8cc"}
Oct 30 21:23:15.669133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.669070    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr" event={"ID":"a7e4ded2-6260-4cb9-942f-99e66504e118","Type":"ContainerStarted","Data":"6ac023dfe34aafd07cc38891f0c9c4d7a63cee491d3b6965cc82b3e1685dee99"}
Oct 30 21:23:15.669133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.669097    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:15.669261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.669139    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:23:15.681081 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.681035    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr" podStartSLOduration=1.681020127 podStartE2EDuration="1.681020127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:15.678965824 +0000 UTC m=+2762.170782371" watchObservedRunningTime="2026-10-30 21:23:15.681020127 +0000 UTC m=+2762.172836675"
Oct 30 21:23:15.731869 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:15.731822    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-plugins-e2e-extmod-a-99668e: configmap "payload-processing-external-model-plugins-e2e-extmod-a-99668e" not found
Oct 30 21:23:15.732053 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:15.731910    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume podName:45f86b98-ce6e-42a5-9177-740f98ab1b17 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:16.231889469 +0000 UTC m=+2762.723705997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-99668e" not found
Oct 30 21:23:15.858813 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:15.854278    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:16.075555 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:16.072980    4810 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b4eff0c43a87: secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:16.075555 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:16.073011    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp: secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:16.075555 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:16.073115    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials podName:a5b6cffb-4d93-40d9-9a81-9ee452617378 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:20.073096275 +0000 UTC m=+2766.564912813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials") pod "payload-processing-external-model-ff5d646d-d5xlp" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378") : secret "agc-cred-b4eff0c43a87" not found
Oct 30 21:23:16.275766 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:16.275691    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-plugins-e2e-extmod-a-99668e: configmap "payload-processing-external-model-plugins-e2e-extmod-a-99668e" not found
Oct 30 21:23:16.276417 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:16.276399    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume podName:45f86b98-ce6e-42a5-9177-740f98ab1b17 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:17.276377404 +0000 UTC m=+2763.768193942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-99668e" not found
Oct 30 21:23:16.276512 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:16.275887    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-aaf84e248242: secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:16.276613 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:16.276603    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z: secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:16.276726 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:16.276715    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials podName:45f86b98-ce6e-42a5-9177-740f98ab1b17 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:18.276700461 +0000 UTC m=+2764.768517001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17") : secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:16.353849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:16.353818    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"]
Oct 30 21:23:16.354170 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:16.354148    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e8ffe3_c3e2_4bc9_8fe4_f49a9945ae28.slice/crio-3917cc7508ed0e314290c22d31756f84650ae3a1adfb44c68f1d25214fe80828 WatchSource:0}: Error finding container 3917cc7508ed0e314290c22d31756f84650ae3a1adfb44c68f1d25214fe80828: Status 404 returned error can't find the container with id 3917cc7508ed0e314290c22d31756f84650ae3a1adfb44c68f1d25214fe80828
Oct 30 21:23:16.671506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:16.671465    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g" event={"ID":"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28","Type":"ContainerStarted","Data":"4ef88276733bd7be535172e7f69c01f2222bae86b217707213f85fa70e145b84"}
Oct 30 21:23:16.671714 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:16.671528    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g" event={"ID":"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28","Type":"ContainerStarted","Data":"3917cc7508ed0e314290c22d31756f84650ae3a1adfb44c68f1d25214fe80828"}
Oct 30 21:23:16.671714 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:16.671558    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:16.693933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:16.693881    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g" podStartSLOduration=1.693867115 podStartE2EDuration="1.693867115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:16.691907543 +0000 UTC m=+2763.183724087" watchObservedRunningTime="2026-10-30 21:23:16.693867115 +0000 UTC m=+2763.185683661"
Oct 30 21:23:17.295566 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:17.295528    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-plugins-e2e-extmod-a-99668e: configmap "payload-processing-external-model-plugins-e2e-extmod-a-99668e" not found
Oct 30 21:23:17.296091 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:17.295626    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume podName:45f86b98-ce6e-42a5-9177-740f98ab1b17 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:19.295605872 +0000 UTC m=+2765.787422414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-99668e" not found
Oct 30 21:23:17.324144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.324091    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f" podUID="946ceaf4-9142-421e-9532-89c096747923" containerName="payload-processing" containerID="cri-o://4195fc8eed58ffeb6c9d88740b1f07cfadafc254ba4a1f2ddc5e37fc9cd87b2b" gracePeriod=25
Oct 30 21:23:17.447954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.447920    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"]
Oct 30 21:23:17.453795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.453733    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.465207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.465182    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"]
Oct 30 21:23:17.500733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.500055    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.500733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.500160    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.500733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.500309    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.500733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.500371    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shcx2\" (UniqueName: \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-kube-api-access-shcx2\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.758913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.757618    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.758913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.757711    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shcx2\" (UniqueName: \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-kube-api-access-shcx2\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.758913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.758023    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.759248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.759053    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.808719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.779922    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.808719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.780629    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.808719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.788577    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.808719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.802623    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shcx2\" (UniqueName: \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-kube-api-access-shcx2\") pod \"payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") " pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:17.831879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.829579    4810 generic.go:356] "Generic (PLEG): container finished" podID="946ceaf4-9142-421e-9532-89c096747923" containerID="4195fc8eed58ffeb6c9d88740b1f07cfadafc254ba4a1f2ddc5e37fc9cd87b2b" exitCode=0
Oct 30 21:23:17.831879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.830682    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f" event={"ID":"946ceaf4-9142-421e-9532-89c096747923","Type":"ContainerDied","Data":"4195fc8eed58ffeb6c9d88740b1f07cfadafc254ba4a1f2ddc5e37fc9cd87b2b"}
Oct 30 21:23:17.863338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.861961    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:23:17.905529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.905494    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-kube-api-access-4hzsb\" (UniqueName: \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-kube-api-access-4hzsb\") pod \"946ceaf4-9142-421e-9532-89c096747923\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") "
Oct 30 21:23:17.905722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.905689    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-routing-overlay-volume\") pod \"946ceaf4-9142-421e-9532-89c096747923\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") "
Oct 30 21:23:17.905909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.905885    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-provider-credentials\" (UniqueName: \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-provider-credentials\") pod \"946ceaf4-9142-421e-9532-89c096747923\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") "
Oct 30 21:23:17.906005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.905971    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-plugins-config-volume\") pod \"946ceaf4-9142-421e-9532-89c096747923\" (UID: \"946ceaf4-9142-421e-9532-89c096747923\") "
Oct 30 21:23:17.906157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.906129    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-routing-overlay-volume" pod "946ceaf4-9142-421e-9532-89c096747923" (UID: "946ceaf4-9142-421e-9532-89c096747923"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:17.906712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.906685    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-plugins-config-volume" pod "946ceaf4-9142-421e-9532-89c096747923" (UID: "946ceaf4-9142-421e-9532-89c096747923"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:17.907354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.907329    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.908224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.907373    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/946ceaf4-9142-421e-9532-89c096747923-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.929512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.929482    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-provider-credentials" pod "946ceaf4-9142-421e-9532-89c096747923" (UID: "946ceaf4-9142-421e-9532-89c096747923"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:17.930459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.930433    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-kube-api-access-4hzsb" pod "946ceaf4-9142-421e-9532-89c096747923" (UID: "946ceaf4-9142-421e-9532-89c096747923"). InnerVolumeSpecName "kube-api-access-4hzsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:17.974807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.974766    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"]
Oct 30 21:23:17.975679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.975657    4810 cpu_manager.go:414] "containerMap: removing container" podUID="946ceaf4-9142-421e-9532-89c096747923" containerName="payload-processing"
Oct 30 21:23:17.975828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.975680    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946ceaf4-9142-421e-9532-89c096747923" containerName="payload-processing"
Oct 30 21:23:17.975975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.975959    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="946ceaf4-9142-421e-9532-89c096747923" containerName="payload-processing"
Oct 30 21:23:17.980367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.980342    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:17.986144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:17.986115    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"]
Oct 30 21:23:18.039936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.037326    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fnld\" (UniqueName: \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-kube-api-access-8fnld\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.039936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.037481    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.039936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.037559    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.039936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.037795    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.039936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.038035    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:18.039936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.038057    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hzsb\" (UniqueName: \"kubernetes.io/projected/946ceaf4-9142-421e-9532-89c096747923-kube-api-access-4hzsb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:18.101879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.100262    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:18.139325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.139272    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fnld\" (UniqueName: \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-kube-api-access-8fnld\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.139639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.139612    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.139874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.139827    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.140202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.140180    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.142401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.142322    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.143424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.143380    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.143671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.143615    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.157066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.157030    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fnld\" (UniqueName: \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-kube-api-access-8fnld\") pod \"payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") " pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.238597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.238561    4810 scope.go:122] "RemoveContainer" containerID="94be1b8e87ab6b720f0e1c8106dee2666e323fb292d4c75b9a52da2ee9376070"
Oct 30 21:23:18.260771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.260749    4810 scope.go:122] "RemoveContainer" containerID="dfde8135d1468102329852696cec9695a15bf6aefb224ad95c08bbb4912abd0f"
Oct 30 21:23:18.346942 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:18.346261    4810 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99668e/agc-cred-aaf84e248242: secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:18.346942 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:18.346293    4810 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z: secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:18.346942 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:18.346352    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials podName:45f86b98-ce6e-42a5-9177-740f98ab1b17 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:22.346333614 +0000 UTC m=+2768.838150152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17") : secret "agc-cred-aaf84e248242" not found
Oct 30 21:23:18.396978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.396659    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:18.416096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.409339    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"]
Oct 30 21:23:18.433732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.433704    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.441730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.441704    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:23:18.442594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.441999    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:18.463067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.461592    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"]
Oct 30 21:23:18.468681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.468656    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"]
Oct 30 21:23:18.475080 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:18.475054    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b2ea22_c592_4df3_aa6d_5108cf2b5094.slice/crio-6b1191572ff77d70389e474bac8193b55bd74f3979c1c3fc78948071b6377558 WatchSource:0}: Error finding container 6b1191572ff77d70389e474bac8193b55bd74f3979c1c3fc78948071b6377558: Status 404 returned error can't find the container with id 6b1191572ff77d70389e474bac8193b55bd74f3979c1c3fc78948071b6377558
Oct 30 21:23:18.559692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.559663    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d407f0c1-bkgcq\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.559930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.559909    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ql4x\" (UniqueName: \"kubernetes.io/projected/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-kube-api-access-4ql4x\") pod \"maas-api-revoke-sub-d407f0c1-bkgcq\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.560112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.560096    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d407f0c1-bkgcq\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.568069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.568038    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp" podUID="a5b6cffb-4d93-40d9-9a81-9ee452617378" containerName="payload-processing" containerID="cri-o://7fa6a40f18ac96cce4dcd0aaf190e31f28690c18f03da80b601288e53cec3ecf" gracePeriod=25
Oct 30 21:23:18.662011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.661966    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ql4x\" (UniqueName: \"kubernetes.io/projected/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-kube-api-access-4ql4x\") pod \"maas-api-revoke-sub-d407f0c1-bkgcq\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.662203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.662146    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d407f0c1-bkgcq\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.662268 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.662207    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d407f0c1-bkgcq\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.662794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.662773    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d407f0c1-bkgcq\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.664649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.664625    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d407f0c1-bkgcq\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.670132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.670112    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ql4x\" (UniqueName: \"kubernetes.io/projected/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-kube-api-access-4ql4x\") pod \"maas-api-revoke-sub-d407f0c1-bkgcq\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.765021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.763130    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:18.788674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.788640    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"]
Oct 30 21:23:18.796258 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:18.796226    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a59394_b598_43d8_a666_415395af7d2e.slice/crio-1e81b58bf07a6073c7e24b4ad14ffe115af0af109108674dd9a3ecd8bd105129 WatchSource:0}: Error finding container 1e81b58bf07a6073c7e24b4ad14ffe115af0af109108674dd9a3ecd8bd105129: Status 404 returned error can't find the container with id 1e81b58bf07a6073c7e24b4ad14ffe115af0af109108674dd9a3ecd8bd105129
Oct 30 21:23:18.852796 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.851546    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9" event={"ID":"96a59394-b598-43d8-a666-415395af7d2e","Type":"ContainerStarted","Data":"1e81b58bf07a6073c7e24b4ad14ffe115af0af109108674dd9a3ecd8bd105129"}
Oct 30 21:23:18.853848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.853142    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5" event={"ID":"86b2ea22-c592-4df3-aa6d-5108cf2b5094","Type":"ContainerStarted","Data":"a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3"}
Oct 30 21:23:18.853848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.853185    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5" event={"ID":"86b2ea22-c592-4df3-aa6d-5108cf2b5094","Type":"ContainerStarted","Data":"6b1191572ff77d70389e474bac8193b55bd74f3979c1c3fc78948071b6377558"}
Oct 30 21:23:18.857923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.854086    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:18.857923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.855873    4810 generic.go:356] "Generic (PLEG): container finished" podID="a5b6cffb-4d93-40d9-9a81-9ee452617378" containerID="7fa6a40f18ac96cce4dcd0aaf190e31f28690c18f03da80b601288e53cec3ecf" exitCode=0
Oct 30 21:23:18.857923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.855954    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp" event={"ID":"a5b6cffb-4d93-40d9-9a81-9ee452617378","Type":"ContainerDied","Data":"7fa6a40f18ac96cce4dcd0aaf190e31f28690c18f03da80b601288e53cec3ecf"}
Oct 30 21:23:18.862085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.858972    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f" event={"ID":"946ceaf4-9142-421e-9532-89c096747923","Type":"ContainerDied","Data":"9b26bd99a4ab9bc4d0cff86748dc939984aa23a87eb6750ba6223e29744019f8"}
Oct 30 21:23:18.862085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.859022    4810 scope.go:122] "RemoveContainer" containerID="4195fc8eed58ffeb6c9d88740b1f07cfadafc254ba4a1f2ddc5e37fc9cd87b2b"
Oct 30 21:23:18.862085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.859149    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"
Oct 30 21:23:18.963977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:18.963944    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:19.023321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.023253    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5" podStartSLOduration=2.023233144 podStartE2EDuration="2.023233144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:18.896425483 +0000 UTC m=+2765.388242029" watchObservedRunningTime="2026-10-30 21:23:19.023233144 +0000 UTC m=+2765.515049691"
Oct 30 21:23:19.066899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.066764    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"]
Oct 30 21:23:19.075917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.074001    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-routing-overlay-volume\") pod \"a5b6cffb-4d93-40d9-9a81-9ee452617378\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") "
Oct 30 21:23:19.075917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.074173    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-kube-api-access-5qzkj\" (UniqueName: \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-kube-api-access-5qzkj\") pod \"a5b6cffb-4d93-40d9-9a81-9ee452617378\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") "
Oct 30 21:23:19.075917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.074216    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials\") pod \"a5b6cffb-4d93-40d9-9a81-9ee452617378\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") "
Oct 30 21:23:19.075917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.074323    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume\") pod \"a5b6cffb-4d93-40d9-9a81-9ee452617378\" (UID: \"a5b6cffb-4d93-40d9-9a81-9ee452617378\") "
Oct 30 21:23:19.075917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.075518    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume" pod "a5b6cffb-4d93-40d9-9a81-9ee452617378" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:19.075917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.075905    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-routing-overlay-volume" pod "a5b6cffb-4d93-40d9-9a81-9ee452617378" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:19.076437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.076414    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-9f8bd7d97-kfm4f"]
Oct 30 21:23:19.079710 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.079689    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials" pod "a5b6cffb-4d93-40d9-9a81-9ee452617378" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:19.079954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.079930    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-kube-api-access-5qzkj" pod "a5b6cffb-4d93-40d9-9a81-9ee452617378" (UID: "a5b6cffb-4d93-40d9-9a81-9ee452617378"). InnerVolumeSpecName "kube-api-access-5qzkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:19.109573 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:19.109539    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede1ea4e_4f15_4e9f_92ca_ca70f9925dff.slice/crio-bd09fa64accc8823ae5805c379cd046ff0b66c776f0cdabc561f8d9e4a97ae61 WatchSource:0}: Error finding container bd09fa64accc8823ae5805c379cd046ff0b66c776f0cdabc561f8d9e4a97ae61: Status 404 returned error can't find the container with id bd09fa64accc8823ae5805c379cd046ff0b66c776f0cdabc561f8d9e4a97ae61
Oct 30 21:23:19.109731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.109683    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"]
Oct 30 21:23:19.177087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.177051    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.177087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.177088    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qzkj\" (UniqueName: \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-kube-api-access-5qzkj\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.177281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.177102    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5b6cffb-4d93-40d9-9a81-9ee452617378-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.177281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.177116    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b6cffb-4d93-40d9-9a81-9ee452617378-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.380380 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:19.380345    4810 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-plugins-e2e-extmod-a-99668e: configmap "payload-processing-external-model-plugins-e2e-extmod-a-99668e" not found
Oct 30 21:23:19.380762 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:19.380414    4810 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume podName:45f86b98-ce6e-42a5-9177-740f98ab1b17 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:23.38040015 +0000 UTC m=+2769.872216673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-99668e" not found
Oct 30 21:23:19.408892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.408843    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"]
Oct 30 21:23:19.409696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.409679    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a5b6cffb-4d93-40d9-9a81-9ee452617378" containerName="payload-processing"
Oct 30 21:23:19.409751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.409697    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b6cffb-4d93-40d9-9a81-9ee452617378" containerName="payload-processing"
Oct 30 21:23:19.409827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.409818    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b6cffb-4d93-40d9-9a81-9ee452617378" containerName="payload-processing"
Oct 30 21:23:19.415294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.415266    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:23:19.419592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.419566    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ddcf93\""
Oct 30 21:23:19.424982 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.424744    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"]
Oct 30 21:23:19.448281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.448239    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"]
Oct 30 21:23:19.453588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.453561    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:23:19.464203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.464168    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"]
Oct 30 21:23:19.481103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.481061    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ddcf93-dockercfg-686dz\""
Oct 30 21:23:19.481729 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.481687    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb523c54-0274-45d9-960d-b395e30205f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24\" (UID: \"cb523c54-0274-45d9-960d-b395e30205f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:23:19.481885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.481729    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shgb\" (UniqueName: \"kubernetes.io/projected/cb523c54-0274-45d9-960d-b395e30205f1-kube-api-access-8shgb\") pod \"payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24\" (UID: \"cb523c54-0274-45d9-960d-b395e30205f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:23:19.586535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.583944    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb523c54-0274-45d9-960d-b395e30205f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24\" (UID: \"cb523c54-0274-45d9-960d-b395e30205f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:23:19.586535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.583994    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shgb\" (UniqueName: \"kubernetes.io/projected/cb523c54-0274-45d9-960d-b395e30205f1-kube-api-access-8shgb\") pod \"payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24\" (UID: \"cb523c54-0274-45d9-960d-b395e30205f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:23:19.586535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.584078    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq65z\" (UniqueName: \"kubernetes.io/projected/95743c31-995c-4ff4-b66c-174478a72987-kube-api-access-qq65z\") pod \"payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk\" (UID: \"95743c31-995c-4ff4-b66c-174478a72987\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:23:19.586535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.584157    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95743c31-995c-4ff4-b66c-174478a72987-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk\" (UID: \"95743c31-995c-4ff4-b66c-174478a72987\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:23:19.586535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.585214    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb523c54-0274-45d9-960d-b395e30205f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24\" (UID: \"cb523c54-0274-45d9-960d-b395e30205f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:23:19.641301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.638054    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shgb\" (UniqueName: \"kubernetes.io/projected/cb523c54-0274-45d9-960d-b395e30205f1-kube-api-access-8shgb\") pod \"payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24\" (UID: \"cb523c54-0274-45d9-960d-b395e30205f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:23:19.674514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.674483    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946ceaf4-9142-421e-9532-89c096747923" path="/var/lib/kubelet/pods/946ceaf4-9142-421e-9532-89c096747923/volumes"
Oct 30 21:23:19.685531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.685474    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq65z\" (UniqueName: \"kubernetes.io/projected/95743c31-995c-4ff4-b66c-174478a72987-kube-api-access-qq65z\") pod \"payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk\" (UID: \"95743c31-995c-4ff4-b66c-174478a72987\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:23:19.685901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.685832    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95743c31-995c-4ff4-b66c-174478a72987-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk\" (UID: \"95743c31-995c-4ff4-b66c-174478a72987\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:23:19.686906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.686852    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95743c31-995c-4ff4-b66c-174478a72987-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk\" (UID: \"95743c31-995c-4ff4-b66c-174478a72987\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:23:19.695720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.695697    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq65z\" (UniqueName: \"kubernetes.io/projected/95743c31-995c-4ff4-b66c-174478a72987-kube-api-access-qq65z\") pod \"payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk\" (UID: \"95743c31-995c-4ff4-b66c-174478a72987\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:23:19.735760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.735722    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:23:19.777228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.777199    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:23:19.877517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.877433    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9" event={"ID":"96a59394-b598-43d8-a666-415395af7d2e","Type":"ContainerStarted","Data":"cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553"}
Oct 30 21:23:19.879849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.879694    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq" event={"ID":"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff","Type":"ContainerStarted","Data":"4bc227132ba6a3df5265584d1dcbfef136c4d10a94e1f6b434fd3a31947664c0"}
Oct 30 21:23:19.879849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.879751    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq" event={"ID":"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff","Type":"ContainerStarted","Data":"bd09fa64accc8823ae5805c379cd046ff0b66c776f0cdabc561f8d9e4a97ae61"}
Oct 30 21:23:19.881642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.881618    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"
Oct 30 21:23:19.882043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.882017    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp" event={"ID":"a5b6cffb-4d93-40d9-9a81-9ee452617378","Type":"ContainerDied","Data":"abb61ec2c83376159e230f1e5f7fc1b24cd0fbe7213ce9b31cc3685156608aba"}
Oct 30 21:23:19.882152 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.882076    4810 scope.go:122] "RemoveContainer" containerID="7fa6a40f18ac96cce4dcd0aaf190e31f28690c18f03da80b601288e53cec3ecf"
Oct 30 21:23:19.898436 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.898348    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9" podStartSLOduration=2.898333633 podStartE2EDuration="2.898333633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:19.897611294 +0000 UTC m=+2766.389427852" watchObservedRunningTime="2026-10-30 21:23:19.898333633 +0000 UTC m=+2766.390150225"
Oct 30 21:23:19.906184 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.906056    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc" podUID="bb4bf0bd-8388-4b15-bed9-679f1dab2da7" containerName="payload-processing" containerID="cri-o://22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89" gracePeriod=25
Oct 30 21:23:19.942435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.942389    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq" podStartSLOduration=1.942375439 podStartE2EDuration="1.942375439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:19.926066641 +0000 UTC m=+2766.417883210" watchObservedRunningTime="2026-10-30 21:23:19.942375439 +0000 UTC m=+2766.434191983"
Oct 30 21:23:19.946964 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.946931    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"]
Oct 30 21:23:19.948675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:19.948651    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5d646d-d5xlp"]
Oct 30 21:23:20.041234 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:20.041189    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb523c54_0274_45d9_960d_b395e30205f1.slice/crio-07fa9e4dde2e7fe7e66566011ced3cc85ccc593cb6017953ec5333f38d6a6a23 WatchSource:0}: Error finding container 07fa9e4dde2e7fe7e66566011ced3cc85ccc593cb6017953ec5333f38d6a6a23: Status 404 returned error can't find the container with id 07fa9e4dde2e7fe7e66566011ced3cc85ccc593cb6017953ec5333f38d6a6a23
Oct 30 21:23:20.042242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.042209    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"]
Oct 30 21:23:20.126071 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:20.126035    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95743c31_995c_4ff4_b66c_174478a72987.slice/crio-98144e32d301799b7b08331264fb8615299d176af4f7bf25446a49fc7a554234 WatchSource:0}: Error finding container 98144e32d301799b7b08331264fb8615299d176af4f7bf25446a49fc7a554234: Status 404 returned error can't find the container with id 98144e32d301799b7b08331264fb8615299d176af4f7bf25446a49fc7a554234
Oct 30 21:23:20.129974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.129482    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"]
Oct 30 21:23:20.259559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.259536    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:23:20.317169 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.317138    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-plugins-config-volume\") pod \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") "
Oct 30 21:23:20.317311 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.317221    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-routing-overlay-volume\") pod \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") "
Oct 30 21:23:20.317401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.317371    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-provider-credentials\") pod \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") "
Oct 30 21:23:20.317466 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.317409    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-kube-api-access-kbk52\" (UniqueName: \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-kube-api-access-kbk52\") pod \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\" (UID: \"bb4bf0bd-8388-4b15-bed9-679f1dab2da7\") "
Oct 30 21:23:20.317584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.317563    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-routing-overlay-volume" pod "bb4bf0bd-8388-4b15-bed9-679f1dab2da7" (UID: "bb4bf0bd-8388-4b15-bed9-679f1dab2da7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:20.317653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.317598    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-plugins-config-volume" pod "bb4bf0bd-8388-4b15-bed9-679f1dab2da7" (UID: "bb4bf0bd-8388-4b15-bed9-679f1dab2da7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:20.318119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.318096    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:20.318210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.318121    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:20.320116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.320092    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-kube-api-access-kbk52" pod "bb4bf0bd-8388-4b15-bed9-679f1dab2da7" (UID: "bb4bf0bd-8388-4b15-bed9-679f1dab2da7"). InnerVolumeSpecName "kube-api-access-kbk52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:20.320219 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.320125    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-provider-credentials" pod "bb4bf0bd-8388-4b15-bed9-679f1dab2da7" (UID: "bb4bf0bd-8388-4b15-bed9-679f1dab2da7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:20.419284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.419211    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:20.419284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.419240    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbk52\" (UniqueName: \"kubernetes.io/projected/bb4bf0bd-8388-4b15-bed9-679f1dab2da7-kube-api-access-kbk52\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:20.585390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.585343    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5" podUID="efa17e4b-b270-4208-bb08-b6ab9910a204" containerName="payload-processing" containerID="cri-o://73b0d0eb6da79f5728f5df264cf051de3826c6c4cd86c9e3403325a3ef13467d" gracePeriod=25
Oct 30 21:23:20.662886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.662813    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr" podUID="a7e4ded2-6260-4cb9-942f-99e66504e118" containerName="payload-processing" containerID="cri-o://a305a2ae512f7d008a57afdb56fdc0576f65c2188ce72cb785a6e3e0d0f6e8cc" gracePeriod=25
Oct 30 21:23:20.727178 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.727141    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"]
Oct 30 21:23:20.899676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.899640    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24" event={"ID":"cb523c54-0274-45d9-960d-b395e30205f1","Type":"ContainerStarted","Data":"3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a"}
Oct 30 21:23:20.899908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.899699    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24" event={"ID":"cb523c54-0274-45d9-960d-b395e30205f1","Type":"ContainerStarted","Data":"07fa9e4dde2e7fe7e66566011ced3cc85ccc593cb6017953ec5333f38d6a6a23"}
Oct 30 21:23:20.900748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.900727    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:23:20.902669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.902642    4810 generic.go:356] "Generic (PLEG): container finished" podID="ede1ea4e-4f15-4e9f-92ca-ca70f9925dff" containerID="4bc227132ba6a3df5265584d1dcbfef136c4d10a94e1f6b434fd3a31947664c0" exitCode=0
Oct 30 21:23:20.902783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.902759    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq" event={"ID":"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff","Type":"ContainerDied","Data":"4bc227132ba6a3df5265584d1dcbfef136c4d10a94e1f6b434fd3a31947664c0"}
Oct 30 21:23:20.905996 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.905970    4810 generic.go:356] "Generic (PLEG): container finished" podID="bb4bf0bd-8388-4b15-bed9-679f1dab2da7" containerID="22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89" exitCode=0
Oct 30 21:23:20.906128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.906001    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc" event={"ID":"bb4bf0bd-8388-4b15-bed9-679f1dab2da7","Type":"ContainerDied","Data":"22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89"}
Oct 30 21:23:20.906128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.906027    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"
Oct 30 21:23:20.906128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.906044    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc" event={"ID":"bb4bf0bd-8388-4b15-bed9-679f1dab2da7","Type":"ContainerDied","Data":"d1821388ae1add8cd66778fbc4be6ea94f7bb272f14541dcf1182b51a297bc88"}
Oct 30 21:23:20.906128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.906071    4810 scope.go:122] "RemoveContainer" containerID="22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89"
Oct 30 21:23:20.910007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.909978    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk" event={"ID":"95743c31-995c-4ff4-b66c-174478a72987","Type":"ContainerStarted","Data":"40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f"}
Oct 30 21:23:20.910107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.910040    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk" event={"ID":"95743c31-995c-4ff4-b66c-174478a72987","Type":"ContainerStarted","Data":"98144e32d301799b7b08331264fb8615299d176af4f7bf25446a49fc7a554234"}
Oct 30 21:23:20.910107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.910071    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:23:20.911971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.911951    4810 generic.go:356] "Generic (PLEG): container finished" podID="a7e4ded2-6260-4cb9-942f-99e66504e118" containerID="a305a2ae512f7d008a57afdb56fdc0576f65c2188ce72cb785a6e3e0d0f6e8cc" exitCode=0
Oct 30 21:23:20.912081 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.912062    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr" event={"ID":"a7e4ded2-6260-4cb9-942f-99e66504e118","Type":"ContainerDied","Data":"a305a2ae512f7d008a57afdb56fdc0576f65c2188ce72cb785a6e3e0d0f6e8cc"}
Oct 30 21:23:20.940250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.914243    4810 generic.go:356] "Generic (PLEG): container finished" podID="efa17e4b-b270-4208-bb08-b6ab9910a204" containerID="73b0d0eb6da79f5728f5df264cf051de3826c6c4cd86c9e3403325a3ef13467d" exitCode=0
Oct 30 21:23:20.940250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.914300    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5" event={"ID":"efa17e4b-b270-4208-bb08-b6ab9910a204","Type":"ContainerDied","Data":"73b0d0eb6da79f5728f5df264cf051de3826c6c4cd86c9e3403325a3ef13467d"}
Oct 30 21:23:20.940250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:20.916045    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:21.063838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.062207    4810 scope.go:122] "RemoveContainer" containerID="22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89"
Oct 30 21:23:21.063838 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:21.063712    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89\": container with ID starting with 22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89 not found: ID does not exist" containerID="22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89"
Oct 30 21:23:21.063838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.063760    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89"} err="failed to get container status \"22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89\": rpc error: code = NotFound desc = could not find container \"22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89\": container with ID starting with 22b192f3350aefd8e6332f605f264929bc6f4c7520517c9da182d56b480acb89 not found: ID does not exist"
Oct 30 21:23:21.093896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.092941    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24" podStartSLOduration=2.092918626 podStartE2EDuration="2.092918626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:20.974509357 +0000 UTC m=+2767.466325923" watchObservedRunningTime="2026-10-30 21:23:21.092918626 +0000 UTC m=+2767.584735402"
Oct 30 21:23:21.106568 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.102289    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk" podStartSLOduration=2.102268482 podStartE2EDuration="2.102268482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:21.046769972 +0000 UTC m=+2767.538586518" watchObservedRunningTime="2026-10-30 21:23:21.102268482 +0000 UTC m=+2767.594085030"
Oct 30 21:23:21.223538 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.223459    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"]
Oct 30 21:23:21.242571 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.241769    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:23:21.245581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.245536    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:21.250965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.250719    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-7b97f8lvc"]
Oct 30 21:23:21.407879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.390362    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-plugins-config-volume\") pod \"a7e4ded2-6260-4cb9-942f-99e66504e118\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") "
Oct 30 21:23:21.407879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.405820    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-routing-overlay-volume\") pod \"efa17e4b-b270-4208-bb08-b6ab9910a204\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") "
Oct 30 21:23:21.407879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.405904    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-kube-api-access-7m22j\" (UniqueName: \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-kube-api-access-7m22j\") pod \"efa17e4b-b270-4208-bb08-b6ab9910a204\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") "
Oct 30 21:23:21.407879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.405975    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-routing-overlay-volume\") pod \"a7e4ded2-6260-4cb9-942f-99e66504e118\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") "
Oct 30 21:23:21.407879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.406151    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-plugins-config-volume\") pod \"efa17e4b-b270-4208-bb08-b6ab9910a204\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") "
Oct 30 21:23:21.407879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.406221    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-kube-api-access-mn489\" (UniqueName: \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-kube-api-access-mn489\") pod \"a7e4ded2-6260-4cb9-942f-99e66504e118\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") "
Oct 30 21:23:21.407879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.406258    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-provider-credentials\") pod \"a7e4ded2-6260-4cb9-942f-99e66504e118\" (UID: \"a7e4ded2-6260-4cb9-942f-99e66504e118\") "
Oct 30 21:23:21.407879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.406356    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-provider-credentials\" (UniqueName: \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-provider-credentials\") pod \"efa17e4b-b270-4208-bb08-b6ab9910a204\" (UID: \"efa17e4b-b270-4208-bb08-b6ab9910a204\") "
Oct 30 21:23:21.408506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.393519    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-plugins-config-volume" pod "a7e4ded2-6260-4cb9-942f-99e66504e118" (UID: "a7e4ded2-6260-4cb9-942f-99e66504e118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:21.408506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.408127    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-routing-overlay-volume" pod "a7e4ded2-6260-4cb9-942f-99e66504e118" (UID: "a7e4ded2-6260-4cb9-942f-99e66504e118"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:21.408506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.408478    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-routing-overlay-volume" pod "efa17e4b-b270-4208-bb08-b6ab9910a204" (UID: "efa17e4b-b270-4208-bb08-b6ab9910a204"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:21.411820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.411788    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-plugins-config-volume" pod "efa17e4b-b270-4208-bb08-b6ab9910a204" (UID: "efa17e4b-b270-4208-bb08-b6ab9910a204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:21.415114 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.414413    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-kube-api-access-7m22j" pod "efa17e4b-b270-4208-bb08-b6ab9910a204" (UID: "efa17e4b-b270-4208-bb08-b6ab9910a204"). InnerVolumeSpecName "kube-api-access-7m22j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:21.415258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.415211    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-provider-credentials" pod "efa17e4b-b270-4208-bb08-b6ab9910a204" (UID: "efa17e4b-b270-4208-bb08-b6ab9910a204"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:21.415530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.415507    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-kube-api-access-mn489" pod "a7e4ded2-6260-4cb9-942f-99e66504e118" (UID: "a7e4ded2-6260-4cb9-942f-99e66504e118"). InnerVolumeSpecName "kube-api-access-mn489". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:21.415870 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.415830    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-provider-credentials" pod "a7e4ded2-6260-4cb9-942f-99e66504e118" (UID: "a7e4ded2-6260-4cb9-942f-99e66504e118"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:21.508770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.508685    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:21.508770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.508726    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:21.508770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.508740    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m22j\" (UniqueName: \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-kube-api-access-7m22j\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:21.508770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.508754    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7e4ded2-6260-4cb9-942f-99e66504e118-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:21.508770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.508768    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa17e4b-b270-4208-bb08-b6ab9910a204-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:21.509302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.508781    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn489\" (UniqueName: \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-kube-api-access-mn489\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:21.509302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.508793    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7e4ded2-6260-4cb9-942f-99e66504e118-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:21.509302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.508806    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efa17e4b-b270-4208-bb08-b6ab9910a204-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:21.665186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.665152    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b6cffb-4d93-40d9-9a81-9ee452617378" path="/var/lib/kubelet/pods/a5b6cffb-4d93-40d9-9a81-9ee452617378/volumes"
Oct 30 21:23:21.665620 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.665606    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4bf0bd-8388-4b15-bed9-679f1dab2da7" path="/var/lib/kubelet/pods/bb4bf0bd-8388-4b15-bed9-679f1dab2da7/volumes"
Oct 30 21:23:21.672873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.672829    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" podUID="45f86b98-ce6e-42a5-9177-740f98ab1b17" containerName="payload-processing" containerID="cri-o://c7ae0f27821e8f02a8c1f5f5fea95d6bbbf5e90068cc7cd07f27dfab61bb07b0" gracePeriod=25
Oct 30 21:23:21.934889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.934759    4810 generic.go:356] "Generic (PLEG): container finished" podID="45f86b98-ce6e-42a5-9177-740f98ab1b17" containerID="c7ae0f27821e8f02a8c1f5f5fea95d6bbbf5e90068cc7cd07f27dfab61bb07b0" exitCode=0
Oct 30 21:23:21.935218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.935140    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" event={"ID":"45f86b98-ce6e-42a5-9177-740f98ab1b17","Type":"ContainerDied","Data":"c7ae0f27821e8f02a8c1f5f5fea95d6bbbf5e90068cc7cd07f27dfab61bb07b0"}
Oct 30 21:23:21.939269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.937253    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr" event={"ID":"a7e4ded2-6260-4cb9-942f-99e66504e118","Type":"ContainerDied","Data":"6ac023dfe34aafd07cc38891f0c9c4d7a63cee491d3b6965cc82b3e1685dee99"}
Oct 30 21:23:21.939269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.937302    4810 scope.go:122] "RemoveContainer" containerID="a305a2ae512f7d008a57afdb56fdc0576f65c2188ce72cb785a6e3e0d0f6e8cc"
Oct 30 21:23:21.939269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.937444    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"
Oct 30 21:23:21.945822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.945796    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"
Oct 30 21:23:21.948902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.946726    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5" event={"ID":"efa17e4b-b270-4208-bb08-b6ab9910a204","Type":"ContainerDied","Data":"83ce835b4e7afe1e1382f9f7193c944763dfe20ec3842019932c1aac5f76f416"}
Oct 30 21:23:21.965073 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.963817    4810 scope.go:122] "RemoveContainer" containerID="73b0d0eb6da79f5728f5df264cf051de3826c6c4cd86c9e3403325a3ef13467d"
Oct 30 21:23:21.984532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.984056    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"]
Oct 30 21:23:21.995705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:21.995680    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e07442/payload-processing-external-model-e2e-extmod-b-e07442-6776vkcfr"]
Oct 30 21:23:22.010108 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.009169    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"]
Oct 30 21:23:22.018911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.017938    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-99fcqxfp5"]
Oct 30 21:23:22.075931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.075529    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:22.215883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.186356    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume\") pod \"45f86b98-ce6e-42a5-9177-740f98ab1b17\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") "
Oct 30 21:23:22.215883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.186414    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-routing-overlay-volume\") pod \"45f86b98-ce6e-42a5-9177-740f98ab1b17\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") "
Oct 30 21:23:22.215883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.186598    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-kube-api-access-69xp7\" (UniqueName: \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-kube-api-access-69xp7\") pod \"45f86b98-ce6e-42a5-9177-740f98ab1b17\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") "
Oct 30 21:23:22.215883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.186735    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials\" (UniqueName: \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials\") pod \"45f86b98-ce6e-42a5-9177-740f98ab1b17\" (UID: \"45f86b98-ce6e-42a5-9177-740f98ab1b17\") "
Oct 30 21:23:22.232063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.231444    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume" pod "45f86b98-ce6e-42a5-9177-740f98ab1b17" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:22.232743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.232718    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-routing-overlay-volume" pod "45f86b98-ce6e-42a5-9177-740f98ab1b17" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:22.239490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.239450    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-kube-api-access-69xp7" pod "45f86b98-ce6e-42a5-9177-740f98ab1b17" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17"). InnerVolumeSpecName "kube-api-access-69xp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:22.247393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.240383    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials" pod "45f86b98-ce6e-42a5-9177-740f98ab1b17" (UID: "45f86b98-ce6e-42a5-9177-740f98ab1b17"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:22.315971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.312072    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.316223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.316202    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45f86b98-ce6e-42a5-9177-740f98ab1b17-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.316329 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.316315    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69xp7\" (UniqueName: \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-kube-api-access-69xp7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.316420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.316407    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45f86b98-ce6e-42a5-9177-740f98ab1b17-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.415751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.415713    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:22.518929 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.518895    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-kube-api-access-4ql4x\" (UniqueName: \"kubernetes.io/projected/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-kube-api-access-4ql4x\") pod \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") "
Oct 30 21:23:22.519719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.518998    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-xks-ca\") pod \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") "
Oct 30 21:23:22.519719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.519092    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-openshift-ca\") pod \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\" (UID: \"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff\") "
Oct 30 21:23:22.519956 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.519930    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-openshift-ca" pod "ede1ea4e-4f15-4e9f-92ca-ca70f9925dff" (UID: "ede1ea4e-4f15-4e9f-92ca-ca70f9925dff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:22.521751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.521723    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-xks-ca" pod "ede1ea4e-4f15-4e9f-92ca-ca70f9925dff" (UID: "ede1ea4e-4f15-4e9f-92ca-ca70f9925dff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:22.521968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.521813    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-kube-api-access-4ql4x" pod "ede1ea4e-4f15-4e9f-92ca-ca70f9925dff" (UID: "ede1ea4e-4f15-4e9f-92ca-ca70f9925dff"). InnerVolumeSpecName "kube-api-access-4ql4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:22.621501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.621409    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ql4x\" (UniqueName: \"kubernetes.io/projected/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-kube-api-access-4ql4x\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.621501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.621461    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.621501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.621478    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.768097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.768058    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"]
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769196    4810 cpu_manager.go:414] "containerMap: removing container" podUID="efa17e4b-b270-4208-bb08-b6ab9910a204" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769224    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa17e4b-b270-4208-bb08-b6ab9910a204" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769247    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ede1ea4e-4f15-4e9f-92ca-ca70f9925dff" containerName="revoke-keys"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769256    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede1ea4e-4f15-4e9f-92ca-ca70f9925dff" containerName="revoke-keys"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769297    4810 cpu_manager.go:414] "containerMap: removing container" podUID="bb4bf0bd-8388-4b15-bed9-679f1dab2da7" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769305    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4bf0bd-8388-4b15-bed9-679f1dab2da7" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769319    4810 cpu_manager.go:414] "containerMap: removing container" podUID="45f86b98-ce6e-42a5-9177-740f98ab1b17" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769327    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f86b98-ce6e-42a5-9177-740f98ab1b17" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769341    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a7e4ded2-6260-4cb9-942f-99e66504e118" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769347    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e4ded2-6260-4cb9-942f-99e66504e118" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769607    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa17e4b-b270-4208-bb08-b6ab9910a204" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769630    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede1ea4e-4f15-4e9f-92ca-ca70f9925dff" containerName="revoke-keys"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769645    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4bf0bd-8388-4b15-bed9-679f1dab2da7" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769654    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f86b98-ce6e-42a5-9177-740f98ab1b17" containerName="payload-processing"
Oct 30 21:23:22.770142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.769662    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e4ded2-6260-4cb9-942f-99e66504e118" containerName="payload-processing"
Oct 30 21:23:22.786645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.786617    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.798897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.797175    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"]
Oct 30 21:23:22.799248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.799226    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lz7pj\""
Oct 30 21:23:22.842041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.842005    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-routing-overlay-volume\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.842288 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.842265    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-plugins-config-volume\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.842646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.842626    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-provider-credentials\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.843110 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.843085    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkf4\" (UniqueName: \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-kube-api-access-wrkf4\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.944773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.944729    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkf4\" (UniqueName: \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-kube-api-access-wrkf4\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.944998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.944921    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-routing-overlay-volume\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.944998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.944986    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-plugins-config-volume\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.945128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.945041    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-provider-credentials\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.945770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.945742    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-routing-overlay-volume\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.945968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.945845    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-plugins-config-volume\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.948301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.948275    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-provider-credentials\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.953955 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.953926    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkf4\" (UniqueName: \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-kube-api-access-wrkf4\") pod \"payload-processing-external-model-5757ff5f4d-bfhxj\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") " pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:22.959351 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.959330    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"
Oct 30 21:23:22.959441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.959350    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq" event={"ID":"ede1ea4e-4f15-4e9f-92ca-ca70f9925dff","Type":"ContainerDied","Data":"bd09fa64accc8823ae5805c379cd046ff0b66c776f0cdabc561f8d9e4a97ae61"}
Oct 30 21:23:22.959441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.959406    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd09fa64accc8823ae5805c379cd046ff0b66c776f0cdabc561f8d9e4a97ae61"
Oct 30 21:23:22.962545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.962525    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"
Oct 30 21:23:22.962640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.962564    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z" event={"ID":"45f86b98-ce6e-42a5-9177-740f98ab1b17","Type":"ContainerDied","Data":"b4546caa4c3d63a8270a1df6db13c5683ace67ffc730ee1e4a30c1723879b445"}
Oct 30 21:23:22.962640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:22.962618    4810 scope.go:122] "RemoveContainer" containerID="c7ae0f27821e8f02a8c1f5f5fea95d6bbbf5e90068cc7cd07f27dfab61bb07b0"
Oct 30 21:23:23.012012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:23.011978    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"]
Oct 30 21:23:23.019708 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:23.019681    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99668e/payload-processing-external-model-e2e-extmod-a-99668e-847fnxq9z"]
Oct 30 21:23:23.154285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:23.154248    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:23.524829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:23.524788    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"]
Oct 30 21:23:23.539159 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:23.539111    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a53f616_e9c4_4c23_8e3e_c9b0fcf1c6f5.slice/crio-2650c21c4a3a2b87a25a4bee2d28fa5fb1fcd1347fd53848ff4802937f753578 WatchSource:0}: Error finding container 2650c21c4a3a2b87a25a4bee2d28fa5fb1fcd1347fd53848ff4802937f753578: Status 404 returned error can't find the container with id 2650c21c4a3a2b87a25a4bee2d28fa5fb1fcd1347fd53848ff4802937f753578
Oct 30 21:23:23.683297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:23.683259    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f86b98-ce6e-42a5-9177-740f98ab1b17" path="/var/lib/kubelet/pods/45f86b98-ce6e-42a5-9177-740f98ab1b17/volumes"
Oct 30 21:23:23.684190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:23.684170    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e4ded2-6260-4cb9-942f-99e66504e118" path="/var/lib/kubelet/pods/a7e4ded2-6260-4cb9-942f-99e66504e118/volumes"
Oct 30 21:23:23.684919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:23.684849    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa17e4b-b270-4208-bb08-b6ab9910a204" path="/var/lib/kubelet/pods/efa17e4b-b270-4208-bb08-b6ab9910a204/volumes"
Oct 30 21:23:23.979533 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:23.979494    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj" event={"ID":"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5","Type":"ContainerStarted","Data":"bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf"}
Oct 30 21:23:23.979752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:23.979559    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj" event={"ID":"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5","Type":"ContainerStarted","Data":"2650c21c4a3a2b87a25a4bee2d28fa5fb1fcd1347fd53848ff4802937f753578"}
Oct 30 21:23:23.979752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:23.979593    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:24.003705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:24.003653    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj" podStartSLOduration=2.003639122 podStartE2EDuration="2.003639122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:23.996489905 +0000 UTC m=+2770.488306452" watchObservedRunningTime="2026-10-30 21:23:24.003639122 +0000 UTC m=+2770.495455666"
Oct 30 21:23:25.158554 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:25.158513    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:23:25.728032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:25.727986    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g" podUID="f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28" containerName="payload-processing" containerID="cri-o://4ef88276733bd7be535172e7f69c01f2222bae86b217707213f85fa70e145b84" gracePeriod=25
Oct 30 21:23:26.003971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.003937    4810 generic.go:356] "Generic (PLEG): container finished" podID="f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28" containerID="4ef88276733bd7be535172e7f69c01f2222bae86b217707213f85fa70e145b84" exitCode=0
Oct 30 21:23:26.004109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.004006    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g" event={"ID":"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28","Type":"ContainerDied","Data":"4ef88276733bd7be535172e7f69c01f2222bae86b217707213f85fa70e145b84"}
Oct 30 21:23:26.004109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.004059    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g" event={"ID":"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28","Type":"ContainerDied","Data":"3917cc7508ed0e314290c22d31756f84650ae3a1adfb44c68f1d25214fe80828"}
Oct 30 21:23:26.004109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.004078    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3917cc7508ed0e314290c22d31756f84650ae3a1adfb44c68f1d25214fe80828"
Oct 30 21:23:26.024091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.024063    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:26.098781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.098751    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-kube-api-access-vp525\" (UniqueName: \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-kube-api-access-vp525\") pod \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") "
Oct 30 21:23:26.098971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.098792    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-provider-credentials\") pod \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") "
Oct 30 21:23:26.098971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.098945    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-plugins-config-volume\") pod \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") "
Oct 30 21:23:26.099104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.098974    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-routing-overlay-volume\") pod \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\" (UID: \"f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28\") "
Oct 30 21:23:26.099366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.099338    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-plugins-config-volume" pod "f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28" (UID: "f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:26.099470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.099399    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-routing-overlay-volume" pod "f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28" (UID: "f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:26.099597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.099580    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:26.099637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.099602    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:26.100837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.100810    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-provider-credentials" pod "f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28" (UID: "f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:26.100970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.100948    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-kube-api-access-vp525" pod "f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28" (UID: "f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28"). InnerVolumeSpecName "kube-api-access-vp525". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:26.201032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.200997    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp525\" (UniqueName: \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-kube-api-access-vp525\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:26.201032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:26.201033    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:27.014621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:27.014580    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"
Oct 30 21:23:27.047802 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:27.046220    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"]
Oct 30 21:23:27.056294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:27.056264    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-kj29g"]
Oct 30 21:23:27.665482 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:27.665448    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28" path="/var/lib/kubelet/pods/f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28/volumes"
Oct 30 21:23:28.272094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:28.272061    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"]
Oct 30 21:23:28.290025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:28.289993    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:29.215090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.215053    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"]
Oct 30 21:23:29.216341 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.216318    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28" containerName="payload-processing"
Oct 30 21:23:29.216417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.216343    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28" containerName="payload-processing"
Oct 30 21:23:29.216573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.216560    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e8ffe3-c3e2-4bc9-8fe4-f49a9945ae28" containerName="payload-processing"
Oct 30 21:23:29.222564 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.222541    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:23:29.226828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.226803    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"]
Oct 30 21:23:29.230915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.230892    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ffce13\""
Oct 30 21:23:29.240099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.239997    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ffce13-dockercfg-6fx99\""
Oct 30 21:23:29.252072 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.252041    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fvrs\" (UniqueName: \"kubernetes.io/projected/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-kube-api-access-2fvrs\") pod \"payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9\" (UID: \"b92d5aab-dbcf-47e0-a2c9-111af9c207c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:23:29.252429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.252140    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9\" (UID: \"b92d5aab-dbcf-47e0-a2c9-111af9c207c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:23:29.254445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.254421    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"]
Oct 30 21:23:29.259145 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.259124    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:23:29.272173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.271622    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"]
Oct 30 21:23:29.333885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.333181    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"]
Oct 30 21:23:29.348878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.346339    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.359891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.352661    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:29.359891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.353293    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:23:29.359891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.355429    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"]
Oct 30 21:23:29.359891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.358280    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b2eef0-0705-463e-9c89-96f86d0c1576-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b\" (UID: \"33b2eef0-0705-463e-9c89-96f86d0c1576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:23:29.359891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.358401    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkdk\" (UniqueName: \"kubernetes.io/projected/33b2eef0-0705-463e-9c89-96f86d0c1576-kube-api-access-fvkdk\") pod \"payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b\" (UID: \"33b2eef0-0705-463e-9c89-96f86d0c1576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:23:29.359891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.358607    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fvrs\" (UniqueName: \"kubernetes.io/projected/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-kube-api-access-2fvrs\") pod \"payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9\" (UID: \"b92d5aab-dbcf-47e0-a2c9-111af9c207c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:23:29.359891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.359753    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9\" (UID: \"b92d5aab-dbcf-47e0-a2c9-111af9c207c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:23:29.360582 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.360534    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9\" (UID: \"b92d5aab-dbcf-47e0-a2c9-111af9c207c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:23:29.490280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.490191    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b2eef0-0705-463e-9c89-96f86d0c1576-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b\" (UID: \"33b2eef0-0705-463e-9c89-96f86d0c1576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:23:29.490280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.490261    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plb44\" (UniqueName: \"kubernetes.io/projected/dd34c707-cc17-4fc8-9222-16f625340ce3-kube-api-access-plb44\") pod \"maas-api-revoke-sub-501a5638-sspns\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.490499 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.490306    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkdk\" (UniqueName: \"kubernetes.io/projected/33b2eef0-0705-463e-9c89-96f86d0c1576-kube-api-access-fvkdk\") pod \"payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b\" (UID: \"33b2eef0-0705-463e-9c89-96f86d0c1576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:23:29.490571 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.490550    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-501a5638-sspns\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.490635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.490623    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-501a5638-sspns\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.490956 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.490911    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fvrs\" (UniqueName: \"kubernetes.io/projected/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-kube-api-access-2fvrs\") pod \"payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9\" (UID: \"b92d5aab-dbcf-47e0-a2c9-111af9c207c6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:23:29.496492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.496468    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b2eef0-0705-463e-9c89-96f86d0c1576-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b\" (UID: \"33b2eef0-0705-463e-9c89-96f86d0c1576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:23:29.511827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.511795    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkdk\" (UniqueName: \"kubernetes.io/projected/33b2eef0-0705-463e-9c89-96f86d0c1576-kube-api-access-fvkdk\") pod \"payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b\" (UID: \"33b2eef0-0705-463e-9c89-96f86d0c1576\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:23:29.547332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.547296    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:23:29.584168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.584139    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:23:29.594101 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.593496    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-501a5638-sspns\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.595954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.593936    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-501a5638-sspns\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.595954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.594682    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plb44\" (UniqueName: \"kubernetes.io/projected/dd34c707-cc17-4fc8-9222-16f625340ce3-kube-api-access-plb44\") pod \"maas-api-revoke-sub-501a5638-sspns\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.595954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.595170    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-501a5638-sspns\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.605653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.605518    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-501a5638-sspns\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.607587 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.607558    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plb44\" (UniqueName: \"kubernetes.io/projected/dd34c707-cc17-4fc8-9222-16f625340ce3-kube-api-access-plb44\") pod \"maas-api-revoke-sub-501a5638-sspns\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.732462 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.732290    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:29.734932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.734904    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"]
Oct 30 21:23:29.740521 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.740475    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:29.767075 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.767050    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"]
Oct 30 21:23:29.798737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.798700    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5d2\" (UniqueName: \"kubernetes.io/projected/cf757219-2452-43fb-9fa9-eeafc55695b6-kube-api-access-wr5d2\") pod \"maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:29.798947 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.798901    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:29.799034 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.798962    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:29.885804 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:29.885772    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92d5aab_dbcf_47e0_a2c9_111af9c207c6.slice/crio-f877e7ced65022f2bcbee54e16350afc545f58c48b409e3084f1c9dd90424bf7 WatchSource:0}: Error finding container f877e7ced65022f2bcbee54e16350afc545f58c48b409e3084f1c9dd90424bf7: Status 404 returned error can't find the container with id f877e7ced65022f2bcbee54e16350afc545f58c48b409e3084f1c9dd90424bf7
Oct 30 21:23:29.886825 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.886794    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"]
Oct 30 21:23:29.900538 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.900504    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5d2\" (UniqueName: \"kubernetes.io/projected/cf757219-2452-43fb-9fa9-eeafc55695b6-kube-api-access-wr5d2\") pod \"maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:29.900688 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.900644    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:29.900688 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.900681    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:29.903162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.903139    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:29.908139 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.908111    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:29.912969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.912940    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5d2\" (UniqueName: \"kubernetes.io/projected/cf757219-2452-43fb-9fa9-eeafc55695b6-kube-api-access-wr5d2\") pod \"maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:29.947164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:29.947138    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"]
Oct 30 21:23:29.954277 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:29.954253    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b2eef0_0705_463e_9c89_96f86d0c1576.slice/crio-7421d7c0bdbe68c15c5bb91f6d332515d94357bc1d4c6bdae85b8af7ab16bf3d WatchSource:0}: Error finding container 7421d7c0bdbe68c15c5bb91f6d332515d94357bc1d4c6bdae85b8af7ab16bf3d: Status 404 returned error can't find the container with id 7421d7c0bdbe68c15c5bb91f6d332515d94357bc1d4c6bdae85b8af7ab16bf3d
Oct 30 21:23:30.034981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.034951    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"]
Oct 30 21:23:30.040704 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.040676    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e22cc6-j5sfv"]
Oct 30 21:23:30.044185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.044154    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9" event={"ID":"b92d5aab-dbcf-47e0-a2c9-111af9c207c6","Type":"ContainerStarted","Data":"743d59e793a1835ec8fe8945d8604c54269fb7db392888046a60c9190a182a46"}
Oct 30 21:23:30.044277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.044214    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9" event={"ID":"b92d5aab-dbcf-47e0-a2c9-111af9c207c6","Type":"ContainerStarted","Data":"f877e7ced65022f2bcbee54e16350afc545f58c48b409e3084f1c9dd90424bf7"}
Oct 30 21:23:30.044277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.044248    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:23:30.046164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.046135    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b" event={"ID":"33b2eef0-0705-463e-9c89-96f86d0c1576","Type":"ContainerStarted","Data":"9d0d3b76b844914770f7f46c2d70f855f9595098e2fb5ede719791adb78b802d"}
Oct 30 21:23:30.046274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.046187    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b" event={"ID":"33b2eef0-0705-463e-9c89-96f86d0c1576","Type":"ContainerStarted","Data":"7421d7c0bdbe68c15c5bb91f6d332515d94357bc1d4c6bdae85b8af7ab16bf3d"}
Oct 30 21:23:30.046274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.046212    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:23:30.059635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.059603    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"]
Oct 30 21:23:30.059635 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:30.059627    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd34c707_cc17_4fc8_9222_16f625340ce3.slice/crio-84e1c5c6bf030e46b4471f68f168f076fed32a8f727975bb17a7c197e7f212f9 WatchSource:0}: Error finding container 84e1c5c6bf030e46b4471f68f168f076fed32a8f727975bb17a7c197e7f212f9: Status 404 returned error can't find the container with id 84e1c5c6bf030e46b4471f68f168f076fed32a8f727975bb17a7c197e7f212f9
Oct 30 21:23:30.063190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.063151    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9" podStartSLOduration=1.063139135 podStartE2EDuration="1.063139135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:30.061946722 +0000 UTC m=+2776.553763279" watchObservedRunningTime="2026-10-30 21:23:30.063139135 +0000 UTC m=+2776.554955676"
Oct 30 21:23:30.074192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.074170    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:30.081221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.081171    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b" podStartSLOduration=1.081152985 podStartE2EDuration="1.081152985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:30.077291575 +0000 UTC m=+2776.569108122" watchObservedRunningTime="2026-10-30 21:23:30.081152985 +0000 UTC m=+2776.572969531"
Oct 30 21:23:30.369837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.369805    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"]
Oct 30 21:23:30.370782 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:30.370757    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf757219_2452_43fb_9fa9_eeafc55695b6.slice/crio-9631cf658550a21f4655e2622f9e88910604ad84705354fba335f4eee29cf7a0 WatchSource:0}: Error finding container 9631cf658550a21f4655e2622f9e88910604ad84705354fba335f4eee29cf7a0: Status 404 returned error can't find the container with id 9631cf658550a21f4655e2622f9e88910604ad84705354fba335f4eee29cf7a0
Oct 30 21:23:30.918418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.918391    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:30.981409 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:30.980968    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"]
Oct 30 21:23:31.056682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:31.056637    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt" event={"ID":"cf757219-2452-43fb-9fa9-eeafc55695b6","Type":"ContainerStarted","Data":"6e3565375bb61481193cbd901134d8693eadbc049c922a4e7e56e1b109150990"}
Oct 30 21:23:31.056917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:31.056711    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt" event={"ID":"cf757219-2452-43fb-9fa9-eeafc55695b6","Type":"ContainerStarted","Data":"9631cf658550a21f4655e2622f9e88910604ad84705354fba335f4eee29cf7a0"}
Oct 30 21:23:31.058488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:31.058454    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns" event={"ID":"dd34c707-cc17-4fc8-9222-16f625340ce3","Type":"ContainerStarted","Data":"d79df4cb045aa5ed767723a79ca545f139d07ab8a3724d4747ccb6f31903d5f9"}
Oct 30 21:23:31.058603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:31.058513    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns" event={"ID":"dd34c707-cc17-4fc8-9222-16f625340ce3","Type":"ContainerStarted","Data":"84e1c5c6bf030e46b4471f68f168f076fed32a8f727975bb17a7c197e7f212f9"}
Oct 30 21:23:31.075904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:31.073504    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt" podStartSLOduration=2.073488192 podStartE2EDuration="2.073488192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:31.071895799 +0000 UTC m=+2777.563712345" watchObservedRunningTime="2026-10-30 21:23:31.073488192 +0000 UTC m=+2777.565304745"
Oct 30 21:23:31.093526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:31.093455    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns" podStartSLOduration=2.093438771 podStartE2EDuration="2.093438771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:31.088822277 +0000 UTC m=+2777.580638813" watchObservedRunningTime="2026-10-30 21:23:31.093438771 +0000 UTC m=+2777.585255316"
Oct 30 21:23:31.665922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:31.665893    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f499fe9-1e19-443a-a38a-e40f00ed5dc1" path="/var/lib/kubelet/pods/3f499fe9-1e19-443a-a38a-e40f00ed5dc1/volumes"
Oct 30 21:23:31.951067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:31.951038    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:31.951244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:31.951106    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:23:32.004947 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:32.004911    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"]
Oct 30 21:23:32.068616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:32.068577    4810 generic.go:356] "Generic (PLEG): container finished" podID="dd34c707-cc17-4fc8-9222-16f625340ce3" containerID="d79df4cb045aa5ed767723a79ca545f139d07ab8a3724d4747ccb6f31903d5f9" exitCode=0
Oct 30 21:23:32.068784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:32.068650    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns" event={"ID":"dd34c707-cc17-4fc8-9222-16f625340ce3","Type":"ContainerDied","Data":"d79df4cb045aa5ed767723a79ca545f139d07ab8a3724d4747ccb6f31903d5f9"}
Oct 30 21:23:32.070501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:32.070476    4810 generic.go:356] "Generic (PLEG): container finished" podID="cf757219-2452-43fb-9fa9-eeafc55695b6" containerID="6e3565375bb61481193cbd901134d8693eadbc049c922a4e7e56e1b109150990" exitCode=0
Oct 30 21:23:32.070644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:32.070559    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt" event={"ID":"cf757219-2452-43fb-9fa9-eeafc55695b6","Type":"ContainerDied","Data":"6e3565375bb61481193cbd901134d8693eadbc049c922a4e7e56e1b109150990"}
Oct 30 21:23:32.970756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:32.970729    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:23:33.273008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.272935    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj" podUID="9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5" containerName="payload-processing" containerID="cri-o://bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf" gracePeriod=25
Oct 30 21:23:33.418592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.418561    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:33.423148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.423121    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:33.486874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.485029    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf757219-2452-43fb-9fa9-eeafc55695b6-kube-api-access-wr5d2\" (UniqueName: \"kubernetes.io/projected/cf757219-2452-43fb-9fa9-eeafc55695b6-kube-api-access-wr5d2\") pod \"cf757219-2452-43fb-9fa9-eeafc55695b6\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") "
Oct 30 21:23:33.486874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.485110    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd34c707-cc17-4fc8-9222-16f625340ce3-kube-api-access-plb44\" (UniqueName: \"kubernetes.io/projected/dd34c707-cc17-4fc8-9222-16f625340ce3-kube-api-access-plb44\") pod \"dd34c707-cc17-4fc8-9222-16f625340ce3\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") "
Oct 30 21:23:33.486874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.485196    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-xks-ca\") pod \"dd34c707-cc17-4fc8-9222-16f625340ce3\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") "
Oct 30 21:23:33.486874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.485341    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-openshift-ca\") pod \"dd34c707-cc17-4fc8-9222-16f625340ce3\" (UID: \"dd34c707-cc17-4fc8-9222-16f625340ce3\") "
Oct 30 21:23:33.486874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.485399    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-openshift-ca\") pod \"cf757219-2452-43fb-9fa9-eeafc55695b6\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") "
Oct 30 21:23:33.486874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.485493    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-xks-ca\") pod \"cf757219-2452-43fb-9fa9-eeafc55695b6\" (UID: \"cf757219-2452-43fb-9fa9-eeafc55695b6\") "
Oct 30 21:23:33.488543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.488318    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-openshift-ca" pod "cf757219-2452-43fb-9fa9-eeafc55695b6" (UID: "cf757219-2452-43fb-9fa9-eeafc55695b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:33.489594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.489568    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-openshift-ca" pod "dd34c707-cc17-4fc8-9222-16f625340ce3" (UID: "dd34c707-cc17-4fc8-9222-16f625340ce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:33.491224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.491198    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd34c707-cc17-4fc8-9222-16f625340ce3-kube-api-access-plb44" pod "dd34c707-cc17-4fc8-9222-16f625340ce3" (UID: "dd34c707-cc17-4fc8-9222-16f625340ce3"). InnerVolumeSpecName "kube-api-access-plb44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:33.494609 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.494345    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-xks-ca" pod "dd34c707-cc17-4fc8-9222-16f625340ce3" (UID: "dd34c707-cc17-4fc8-9222-16f625340ce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:33.494722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.494635    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-xks-ca" pod "cf757219-2452-43fb-9fa9-eeafc55695b6" (UID: "cf757219-2452-43fb-9fa9-eeafc55695b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:33.499678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.499656    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf757219-2452-43fb-9fa9-eeafc55695b6-kube-api-access-wr5d2" pod "cf757219-2452-43fb-9fa9-eeafc55695b6" (UID: "cf757219-2452-43fb-9fa9-eeafc55695b6"). InnerVolumeSpecName "kube-api-access-wr5d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:33.566153 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.566128    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:33.586753 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.586723    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-plugins-config-volume\") pod \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") "
Oct 30 21:23:33.586894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.586830    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-kube-api-access-wrkf4\" (UniqueName: \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-kube-api-access-wrkf4\") pod \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") "
Oct 30 21:23:33.586970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.586942    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-routing-overlay-volume\") pod \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") "
Oct 30 21:23:33.587127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587101    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-plugins-config-volume" pod "9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5" (UID: "9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:33.587222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587133    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-provider-credentials\") pod \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\" (UID: \"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5\") "
Oct 30 21:23:33.587320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587295    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-routing-overlay-volume" pod "9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5" (UID: "9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:33.587873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587838    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:33.587987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587888    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd34c707-cc17-4fc8-9222-16f625340ce3-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:33.587987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587900    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:33.587987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587910    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf757219-2452-43fb-9fa9-eeafc55695b6-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:33.587987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587919    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:33.587987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587928    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5d2\" (UniqueName: \"kubernetes.io/projected/cf757219-2452-43fb-9fa9-eeafc55695b6-kube-api-access-wr5d2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:33.587987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587941    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plb44\" (UniqueName: \"kubernetes.io/projected/dd34c707-cc17-4fc8-9222-16f625340ce3-kube-api-access-plb44\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:33.587987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.587949    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:33.590194 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.589929    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-kube-api-access-wrkf4" pod "9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5" (UID: "9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5"). InnerVolumeSpecName "kube-api-access-wrkf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:33.590748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.590705    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-provider-credentials" pod "9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5" (UID: "9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:33.689055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.689024    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkf4\" (UniqueName: \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-kube-api-access-wrkf4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:33.689055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:33.689055    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:34.092444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.092406    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"
Oct 30 21:23:34.092919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.092460    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt" event={"ID":"cf757219-2452-43fb-9fa9-eeafc55695b6","Type":"ContainerDied","Data":"9631cf658550a21f4655e2622f9e88910604ad84705354fba335f4eee29cf7a0"}
Oct 30 21:23:34.092919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.092513    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9631cf658550a21f4655e2622f9e88910604ad84705354fba335f4eee29cf7a0"
Oct 30 21:23:34.094490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.094459    4810 generic.go:356] "Generic (PLEG): container finished" podID="9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5" containerID="bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf" exitCode=0
Oct 30 21:23:34.094607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.094505    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"
Oct 30 21:23:34.094607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.094549    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj" event={"ID":"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5","Type":"ContainerDied","Data":"bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf"}
Oct 30 21:23:34.094607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.094590    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj" event={"ID":"9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5","Type":"ContainerDied","Data":"2650c21c4a3a2b87a25a4bee2d28fa5fb1fcd1347fd53848ff4802937f753578"}
Oct 30 21:23:34.094907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.094618    4810 scope.go:122] "RemoveContainer" containerID="bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf"
Oct 30 21:23:34.097142 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.097119    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns" event={"ID":"dd34c707-cc17-4fc8-9222-16f625340ce3","Type":"ContainerDied","Data":"84e1c5c6bf030e46b4471f68f168f076fed32a8f727975bb17a7c197e7f212f9"}
Oct 30 21:23:34.097217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.097147    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"
Oct 30 21:23:34.097289 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.097170    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84e1c5c6bf030e46b4471f68f168f076fed32a8f727975bb17a7c197e7f212f9"
Oct 30 21:23:34.109555 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.109447    4810 scope.go:122] "RemoveContainer" containerID="bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf"
Oct 30 21:23:34.109812 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:34.109784    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf\": container with ID starting with bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf not found: ID does not exist" containerID="bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf"
Oct 30 21:23:34.109930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.109822    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf"} err="failed to get container status \"bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf\": rpc error: code = NotFound desc = could not find container \"bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf\": container with ID starting with bc9fa5cb18a4a26abc244e83bc0794ab846dabb907999982694f49196471aabf not found: ID does not exist"
Oct 30 21:23:34.125296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.125273    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"]
Oct 30 21:23:34.129641 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:34.129619    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5757ff5f4d-bfhxj"]
Oct 30 21:23:35.665411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:35.665381    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5" path="/var/lib/kubelet/pods/9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5/volumes"
Oct 30 21:23:35.982788 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:35.982679    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8" podUID="5cdb899c-0580-4942-9573-24134c4cc035" containerName="payload-processing" containerID="cri-o://d60e95d5f1f5fd6cc5744f7a6d2aed0ae442e516e97a87c9ae74dc70217599ea" gracePeriod=25
Oct 30 21:23:36.041151 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:36.041124    4810 logging.go:55] [core] [Channel #8346 SubChannel #8348]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.78:50052", ServerName: "10.132.1.78:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.78:50052: connect: connection refused"
Oct 30 21:23:36.168138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.167519    4810 generic.go:356] "Generic (PLEG): container finished" podID="5cdb899c-0580-4942-9573-24134c4cc035" containerID="d60e95d5f1f5fd6cc5744f7a6d2aed0ae442e516e97a87c9ae74dc70217599ea" exitCode=0
Oct 30 21:23:36.168138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.167721    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8" event={"ID":"5cdb899c-0580-4942-9573-24134c4cc035","Type":"ContainerDied","Data":"d60e95d5f1f5fd6cc5744f7a6d2aed0ae442e516e97a87c9ae74dc70217599ea"}
Oct 30 21:23:36.338793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.338767    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:23:36.456228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.456194    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-routing-overlay-volume\") pod \"5cdb899c-0580-4942-9573-24134c4cc035\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") "
Oct 30 21:23:36.456404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.456326    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-kube-api-access-p9d2n\" (UniqueName: \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-kube-api-access-p9d2n\") pod \"5cdb899c-0580-4942-9573-24134c4cc035\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") "
Oct 30 21:23:36.456404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.456355    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-provider-credentials\") pod \"5cdb899c-0580-4942-9573-24134c4cc035\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") "
Oct 30 21:23:36.456404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.456384    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-plugins-config-volume\") pod \"5cdb899c-0580-4942-9573-24134c4cc035\" (UID: \"5cdb899c-0580-4942-9573-24134c4cc035\") "
Oct 30 21:23:36.456637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.456606    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-routing-overlay-volume" pod "5cdb899c-0580-4942-9573-24134c4cc035" (UID: "5cdb899c-0580-4942-9573-24134c4cc035"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:36.456874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.456836    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-plugins-config-volume" pod "5cdb899c-0580-4942-9573-24134c4cc035" (UID: "5cdb899c-0580-4942-9573-24134c4cc035"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:36.457081 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.457063    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:36.457168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.457082    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5cdb899c-0580-4942-9573-24134c4cc035-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:36.458432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.458410    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-kube-api-access-p9d2n" pod "5cdb899c-0580-4942-9573-24134c4cc035" (UID: "5cdb899c-0580-4942-9573-24134c4cc035"). InnerVolumeSpecName "kube-api-access-p9d2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:36.458507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.458440    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-provider-credentials" pod "5cdb899c-0580-4942-9573-24134c4cc035" (UID: "5cdb899c-0580-4942-9573-24134c4cc035"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:36.558445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.558360    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9d2n\" (UniqueName: \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-kube-api-access-p9d2n\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:36.558445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:36.558403    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5cdb899c-0580-4942-9573-24134c4cc035-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:37.016114 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.016064    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b" podUID="5be8a820-9784-4bde-acf6-9a21ea4d0865" containerName="payload-processing" containerID="cri-o://834e7855557a13acad195819312bb492c99230f76665e43b6dc1dea1239da02e" gracePeriod=25
Oct 30 21:23:37.041603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.041570    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8" podUID="5cdb899c-0580-4942-9573-24134c4cc035" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.78:50052\" within 1s: context deadline exceeded"
Oct 30 21:23:37.178003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.177966    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"
Oct 30 21:23:37.178353 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.178321    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8" event={"ID":"5cdb899c-0580-4942-9573-24134c4cc035","Type":"ContainerDied","Data":"d0f982f8fa94f39330f1cf9c055e5bd0c6831ed4391e4c723c09aec4bb9429c0"}
Oct 30 21:23:37.178443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.178410    4810 scope.go:122] "RemoveContainer" containerID="d60e95d5f1f5fd6cc5744f7a6d2aed0ae442e516e97a87c9ae74dc70217599ea"
Oct 30 21:23:37.180389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.180349    4810 generic.go:356] "Generic (PLEG): container finished" podID="5be8a820-9784-4bde-acf6-9a21ea4d0865" containerID="834e7855557a13acad195819312bb492c99230f76665e43b6dc1dea1239da02e" exitCode=0
Oct 30 21:23:37.180528 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.180462    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b" event={"ID":"5be8a820-9784-4bde-acf6-9a21ea4d0865","Type":"ContainerDied","Data":"834e7855557a13acad195819312bb492c99230f76665e43b6dc1dea1239da02e"}
Oct 30 21:23:37.211047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.211009    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"]
Oct 30 21:23:37.220894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.220850    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-7b85zc4c8"]
Oct 30 21:23:37.308293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.308262    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:23:37.369432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.369397    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-provider-credentials\") pod \"5be8a820-9784-4bde-acf6-9a21ea4d0865\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") "
Oct 30 21:23:37.369605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.369591    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-kube-api-access-hn82x\" (UniqueName: \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-kube-api-access-hn82x\") pod \"5be8a820-9784-4bde-acf6-9a21ea4d0865\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") "
Oct 30 21:23:37.369680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.369654    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-plugins-config-volume\") pod \"5be8a820-9784-4bde-acf6-9a21ea4d0865\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") "
Oct 30 21:23:37.369775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.369749    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-routing-overlay-volume\") pod \"5be8a820-9784-4bde-acf6-9a21ea4d0865\" (UID: \"5be8a820-9784-4bde-acf6-9a21ea4d0865\") "
Oct 30 21:23:37.370108 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.370080    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-plugins-config-volume" pod "5be8a820-9784-4bde-acf6-9a21ea4d0865" (UID: "5be8a820-9784-4bde-acf6-9a21ea4d0865"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:37.370108 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.370096    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-routing-overlay-volume" pod "5be8a820-9784-4bde-acf6-9a21ea4d0865" (UID: "5be8a820-9784-4bde-acf6-9a21ea4d0865"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:37.370465 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.370443    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:37.370465 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.370466    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be8a820-9784-4bde-acf6-9a21ea4d0865-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:37.372103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.372074    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-kube-api-access-hn82x" pod "5be8a820-9784-4bde-acf6-9a21ea4d0865" (UID: "5be8a820-9784-4bde-acf6-9a21ea4d0865"). InnerVolumeSpecName "kube-api-access-hn82x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:37.372218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.372115    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-provider-credentials" pod "5be8a820-9784-4bde-acf6-9a21ea4d0865" (UID: "5be8a820-9784-4bde-acf6-9a21ea4d0865"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:37.471811 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.471775    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:37.471811 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.471806    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn82x\" (UniqueName: \"kubernetes.io/projected/5be8a820-9784-4bde-acf6-9a21ea4d0865-kube-api-access-hn82x\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:37.663670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:37.663639    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cdb899c-0580-4942-9573-24134c4cc035" path="/var/lib/kubelet/pods/5cdb899c-0580-4942-9573-24134c4cc035/volumes"
Oct 30 21:23:38.190540 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:38.190508    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"
Oct 30 21:23:38.191019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:38.190539    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b" event={"ID":"5be8a820-9784-4bde-acf6-9a21ea4d0865","Type":"ContainerDied","Data":"fcb08092a18cb2ccc837c0da9d4ff80be2b3864c95e95b3a6a30b5e169b5762b"}
Oct 30 21:23:38.191019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:38.190605    4810 scope.go:122] "RemoveContainer" containerID="834e7855557a13acad195819312bb492c99230f76665e43b6dc1dea1239da02e"
Oct 30 21:23:38.214032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:38.214008    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"]
Oct 30 21:23:38.219079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:38.219058    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-748bgtl6b"]
Oct 30 21:23:38.284900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:38.284877    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:23:39.666248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:39.666217    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be8a820-9784-4bde-acf6-9a21ea4d0865" path="/var/lib/kubelet/pods/5be8a820-9784-4bde-acf6-9a21ea4d0865/volumes"
Oct 30 21:23:40.177505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.177469    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"]
Oct 30 21:23:40.178526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178503    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5be8a820-9784-4bde-acf6-9a21ea4d0865" containerName="payload-processing"
Oct 30 21:23:40.178526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178524    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be8a820-9784-4bde-acf6-9a21ea4d0865" containerName="payload-processing"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178537    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cf757219-2452-43fb-9fa9-eeafc55695b6" containerName="revoke-keys"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178543    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf757219-2452-43fb-9fa9-eeafc55695b6" containerName="revoke-keys"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178561    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5" containerName="payload-processing"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178566    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5" containerName="payload-processing"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178583    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5cdb899c-0580-4942-9573-24134c4cc035" containerName="payload-processing"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178588    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cdb899c-0580-4942-9573-24134c4cc035" containerName="payload-processing"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178604    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dd34c707-cc17-4fc8-9222-16f625340ce3" containerName="revoke-keys"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178608    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd34c707-cc17-4fc8-9222-16f625340ce3" containerName="revoke-keys"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178716    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be8a820-9784-4bde-acf6-9a21ea4d0865" containerName="payload-processing"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178723    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf757219-2452-43fb-9fa9-eeafc55695b6" containerName="revoke-keys"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178732    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a53f616-e9c4-4c23-8e3e-c9b0fcf1c6f5" containerName="payload-processing"
Oct 30 21:23:40.178735 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178741    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cdb899c-0580-4942-9573-24134c4cc035" containerName="payload-processing"
Oct 30 21:23:40.179427 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.178749    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd34c707-cc17-4fc8-9222-16f625340ce3" containerName="revoke-keys"
Oct 30 21:23:40.184499 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.184406    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.194716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.194691    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ddcf93-serving-cert\""
Oct 30 21:23:40.281973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.281938    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"]
Oct 30 21:23:40.323671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.323635    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-metrics-certs\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.323951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.323752    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8445b211-0a2f-484d-bfe8-caf789fd035b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.323951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.323918    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdvl2\" (UniqueName: \"kubernetes.io/projected/8445b211-0a2f-484d-bfe8-caf789fd035b-kube-api-access-hdvl2\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.324144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.323979    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.331658 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.331627    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"]
Oct 30 21:23:40.348192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.348166    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"]
Oct 30 21:23:40.428154 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.428072    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8445b211-0a2f-484d-bfe8-caf789fd035b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.429160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.429131    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdvl2\" (UniqueName: \"kubernetes.io/projected/8445b211-0a2f-484d-bfe8-caf789fd035b-kube-api-access-hdvl2\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.429771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.429745    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.430033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.430012    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-metrics-certs\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.431345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.431320    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8445b211-0a2f-484d-bfe8-caf789fd035b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.456906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.442329    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-metrics-certs\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.456906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.447348    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdvl2\" (UniqueName: \"kubernetes.io/projected/8445b211-0a2f-484d-bfe8-caf789fd035b-kube-api-access-hdvl2\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.483008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.457701    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:40.545904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:40.544963    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:41.108893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:41.104045    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:23:41.236020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:41.232390    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:23:41.238445 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:41.238393    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8445b211_0a2f_484d_bfe8_caf789fd035b.slice/crio-25f089a53cf9b13e662f8accc7529c547ffb9f7df45144758546118376debd02 WatchSource:0}: Error finding container 25f089a53cf9b13e662f8accc7529c547ffb9f7df45144758546118376debd02: Status 404 returned error can't find the container with id 25f089a53cf9b13e662f8accc7529c547ffb9f7df45144758546118376debd02
Oct 30 21:23:41.242975 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:41.239780    4810 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:23:41.255160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:41.255138    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"]
Oct 30 21:23:41.370967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:41.370832    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q" event={"ID":"8445b211-0a2f-484d-bfe8-caf789fd035b","Type":"ContainerStarted","Data":"25f089a53cf9b13e662f8accc7529c547ffb9f7df45144758546118376debd02"}
Oct 30 21:23:41.930960 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:41.930913    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"]
Oct 30 21:23:41.961928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:41.961893    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"]
Oct 30 21:23:42.090268 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.090235    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"]
Oct 30 21:23:42.090603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.090582    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.105399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.105373    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-99668e-openshift-default-dockercfg-442m8\""
Oct 30 21:23:42.231186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.205411    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.231186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.205480    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-envoy\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.231186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.205570    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-certs\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.231186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.205664    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84725bf3-ddbc-49b1-91aa-602ab24f206f-istiod-ca-cert\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.231186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.205713    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-podinfo\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.231186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.205742    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk64b\" (UniqueName: \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-kube-api-access-bk64b\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.231186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.205812    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-credential-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.231186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.205875    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-data\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.231186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.205926    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-token\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.310738    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84725bf3-ddbc-49b1-91aa-602ab24f206f-istiod-ca-cert\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.310816    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-podinfo\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.310850    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk64b\" (UniqueName: \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-kube-api-access-bk64b\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.312599    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84725bf3-ddbc-49b1-91aa-602ab24f206f-istiod-ca-cert\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.313169    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-credential-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.313221    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-data\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.313269    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-token\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.313343    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.313388    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-envoy\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.313474    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-certs\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.313926    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-certs\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.314180    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-credential-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.314401    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-data\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.317883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.314764    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.416893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.411992    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-podinfo\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.416893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.416691    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-token\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.426119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.426060    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-envoy\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.456024 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.455940    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk64b\" (UniqueName: \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-kube-api-access-bk64b\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:42.577881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:42.575091    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:43.059362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:43.059323    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"]
Oct 30 21:23:43.493924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:43.487068    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q" event={"ID":"8445b211-0a2f-484d-bfe8-caf789fd035b","Type":"ContainerStarted","Data":"a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741"}
Oct 30 21:23:43.499419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:43.494777    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:43.522876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:43.515181    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"]
Oct 30 21:23:43.523163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:43.521508    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:43.523385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:43.523361    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:43.523549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:43.523532    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:44.529301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:44.529246    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" event={"ID":"84725bf3-ddbc-49b1-91aa-602ab24f206f","Type":"ContainerStarted","Data":"115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06"}
Oct 30 21:23:44.529780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:44.529330    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" event={"ID":"84725bf3-ddbc-49b1-91aa-602ab24f206f","Type":"ContainerStarted","Data":"9869151ddf7e85d92d202537d00a000ac7861fa5269ba9d95e4ce9c8c7af4f48"}
Oct 30 21:23:44.553622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:44.553515    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" podStartSLOduration=3.553497402 podStartE2EDuration="3.553497402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:44.54975531 +0000 UTC m=+2791.041571856" watchObservedRunningTime="2026-10-30 21:23:44.553497402 +0000 UTC m=+2791.045313947"
Oct 30 21:23:44.557522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:44.557463    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q" podStartSLOduration=3.691040169 podStartE2EDuration="4.557444444s" totalImagesPullingTime="866.404275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:43.54125059 +0000 UTC m=+2790.033067136" watchObservedRunningTime="2026-10-30 21:23:44.557444444 +0000 UTC m=+2791.049260990"
Oct 30 21:23:44.619432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:44.619335    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:44.620871 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:44.620822    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:23:44.621069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:44.621047    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 30 21:23:45.331947 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.331891    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml" podUID="ebf45ae6-8684-4f2d-86a2-fa2d6930f786" containerName="payload-processing" containerID="cri-o://037e07ba7ccc196e52b22b4b9613be23034049b0e4791ccde94b54a5a1d26932" gracePeriod=25
Oct 30 21:23:45.348480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.348428    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4" podUID="3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e" containerName="payload-pre-processing" containerID="cri-o://692a21474f02aac3d63a261cf803282601e2f35886cd6b0e2a9694dd2b7c353d" gracePeriod=25
Oct 30 21:23:45.544974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.544941    4810 generic.go:356] "Generic (PLEG): container finished" podID="3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e" containerID="692a21474f02aac3d63a261cf803282601e2f35886cd6b0e2a9694dd2b7c353d" exitCode=0
Oct 30 21:23:45.545442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.544985    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4" event={"ID":"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e","Type":"ContainerDied","Data":"692a21474f02aac3d63a261cf803282601e2f35886cd6b0e2a9694dd2b7c353d"}
Oct 30 21:23:45.550808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.547346    4810 generic.go:356] "Generic (PLEG): container finished" podID="ebf45ae6-8684-4f2d-86a2-fa2d6930f786" containerID="037e07ba7ccc196e52b22b4b9613be23034049b0e4791ccde94b54a5a1d26932" exitCode=0
Oct 30 21:23:45.550808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.548083    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml" event={"ID":"ebf45ae6-8684-4f2d-86a2-fa2d6930f786","Type":"ContainerDied","Data":"037e07ba7ccc196e52b22b4b9613be23034049b0e4791ccde94b54a5a1d26932"}
Oct 30 21:23:45.617064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.615457    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:23:45.617064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.615516    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 30 21:23:45.837283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.837256    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:23:45.861625 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.861593    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-kube-api-access-tgqvx\" (UniqueName: \"kubernetes.io/projected/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-kube-api-access-tgqvx\") pod \"ebf45ae6-8684-4f2d-86a2-fa2d6930f786\" (UID: \"ebf45ae6-8684-4f2d-86a2-fa2d6930f786\") "
Oct 30 21:23:45.862027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.861940    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-plugins-config-volume\") pod \"ebf45ae6-8684-4f2d-86a2-fa2d6930f786\" (UID: \"ebf45ae6-8684-4f2d-86a2-fa2d6930f786\") "
Oct 30 21:23:45.862398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.862360    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-plugins-config-volume" pod "ebf45ae6-8684-4f2d-86a2-fa2d6930f786" (UID: "ebf45ae6-8684-4f2d-86a2-fa2d6930f786"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:45.863391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.863372    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:23:45.864429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.864408    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-kube-api-access-tgqvx" pod "ebf45ae6-8684-4f2d-86a2-fa2d6930f786" (UID: "ebf45ae6-8684-4f2d-86a2-fa2d6930f786"). InnerVolumeSpecName "kube-api-access-tgqvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:45.963543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.963511    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-plugins-config-volume\") pod \"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e\" (UID: \"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e\") "
Oct 30 21:23:45.963724 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.963550    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-kube-api-access-2k4fd\" (UniqueName: \"kubernetes.io/projected/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-kube-api-access-2k4fd\") pod \"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e\" (UID: \"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e\") "
Oct 30 21:23:45.963977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.963952    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-plugins-config-volume" pod "3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e" (UID: "3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:45.964244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.964225    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:45.964313 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.964250    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgqvx\" (UniqueName: \"kubernetes.io/projected/ebf45ae6-8684-4f2d-86a2-fa2d6930f786-kube-api-access-tgqvx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:45.964313 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.964267    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:45.966035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:45.966008    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-kube-api-access-2k4fd" pod "3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e" (UID: "3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e"). InnerVolumeSpecName "kube-api-access-2k4fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:46.066107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.066073    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k4fd\" (UniqueName: \"kubernetes.io/projected/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e-kube-api-access-2k4fd\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:46.567931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.567900    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"
Oct 30 21:23:46.568414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.567918    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4" event={"ID":"3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e","Type":"ContainerDied","Data":"7c8f23c2eacb9a890d58eda219fe0c24b18320fdcd06adf63235c70e6b70b930"}
Oct 30 21:23:46.568414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.567994    4810 scope.go:122] "RemoveContainer" containerID="692a21474f02aac3d63a261cf803282601e2f35886cd6b0e2a9694dd2b7c353d"
Oct 30 21:23:46.603950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.598741    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:23:46.603950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.598811    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 30 21:23:46.603950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.600158    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"
Oct 30 21:23:46.603950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.600204    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml" event={"ID":"ebf45ae6-8684-4f2d-86a2-fa2d6930f786","Type":"ContainerDied","Data":"58ed79ac95889ad58553e4bf55924c11eb18928aa69f1bf0aed2774ab34c2307"}
Oct 30 21:23:46.622965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.622937    4810 scope.go:122] "RemoveContainer" containerID="037e07ba7ccc196e52b22b4b9613be23034049b0e4791ccde94b54a5a1d26932"
Oct 30 21:23:46.633724 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.633696    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"]
Oct 30 21:23:46.638943 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.638921    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-345bc4-75c8dc5f7f-cnrd4"]
Oct 30 21:23:46.650936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.650911    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"]
Oct 30 21:23:46.656953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.656930    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-345bc4-7555945897-88pml"]
Oct 30 21:23:46.932067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:46.932023    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5" podUID="86b2ea22-c592-4df3-aa6d-5108cf2b5094" containerName="payload-processing" containerID="cri-o://a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3" gracePeriod=25
Oct 30 21:23:47.320750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.320722    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:47.444095 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.444056    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-provider-credentials\" (UniqueName: \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-provider-credentials\") pod \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") "
Oct 30 21:23:47.444286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.444156    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-routing-overlay-volume\") pod \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") "
Oct 30 21:23:47.444286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.444207    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-plugins-config-volume\") pod \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") "
Oct 30 21:23:47.444451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.444321    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-kube-api-access-shcx2\" (UniqueName: \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-kube-api-access-shcx2\") pod \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\" (UID: \"86b2ea22-c592-4df3-aa6d-5108cf2b5094\") "
Oct 30 21:23:47.446498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.445046    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-routing-overlay-volume" pod "86b2ea22-c592-4df3-aa6d-5108cf2b5094" (UID: "86b2ea22-c592-4df3-aa6d-5108cf2b5094"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:47.446498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.445274    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-plugins-config-volume" pod "86b2ea22-c592-4df3-aa6d-5108cf2b5094" (UID: "86b2ea22-c592-4df3-aa6d-5108cf2b5094"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:47.447692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.447214    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-provider-credentials" pod "86b2ea22-c592-4df3-aa6d-5108cf2b5094" (UID: "86b2ea22-c592-4df3-aa6d-5108cf2b5094"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:47.447692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.447628    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-kube-api-access-shcx2" pod "86b2ea22-c592-4df3-aa6d-5108cf2b5094" (UID: "86b2ea22-c592-4df3-aa6d-5108cf2b5094"). InnerVolumeSpecName "kube-api-access-shcx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:47.545941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.545806    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shcx2\" (UniqueName: \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-kube-api-access-shcx2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:47.545941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.545847    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86b2ea22-c592-4df3-aa6d-5108cf2b5094-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:47.545941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.545876    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:47.545941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.545889    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b2ea22-c592-4df3-aa6d-5108cf2b5094-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:47.603879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.603372    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:23:47.603879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.603450    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 30 21:23:47.616220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.615916    4810 generic.go:356] "Generic (PLEG): container finished" podID="86b2ea22-c592-4df3-aa6d-5108cf2b5094" containerID="a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3" exitCode=0
Oct 30 21:23:47.616220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.616036    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"
Oct 30 21:23:47.616220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.616044    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5" event={"ID":"86b2ea22-c592-4df3-aa6d-5108cf2b5094","Type":"ContainerDied","Data":"a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3"}
Oct 30 21:23:47.616220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.616137    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5" event={"ID":"86b2ea22-c592-4df3-aa6d-5108cf2b5094","Type":"ContainerDied","Data":"6b1191572ff77d70389e474bac8193b55bd74f3979c1c3fc78948071b6377558"}
Oct 30 21:23:47.616220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.616165    4810 scope.go:122] "RemoveContainer" containerID="a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3"
Oct 30 21:23:47.629562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.629532    4810 scope.go:122] "RemoveContainer" containerID="a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3"
Oct 30 21:23:47.629870 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:47.629835    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3\": container with ID starting with a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3 not found: ID does not exist" containerID="a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3"
Oct 30 21:23:47.629986 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.629877    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3"} err="failed to get container status \"a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3\": rpc error: code = NotFound desc = could not find container \"a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3\": container with ID starting with a65d2f6e971eecf1946247072dbf5c58475e679949e3187bcc78cfe044fbabd3 not found: ID does not exist"
Oct 30 21:23:47.645342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.645316    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"]
Oct 30 21:23:47.649330 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.649309    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e8063c/payload-processing-external-model-e2e-extmod-a-e8063c-69cfjmfn5"]
Oct 30 21:23:47.662977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.662955    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e" path="/var/lib/kubelet/pods/3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e/volumes"
Oct 30 21:23:47.663372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.663358    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b2ea22-c592-4df3-aa6d-5108cf2b5094" path="/var/lib/kubelet/pods/86b2ea22-c592-4df3-aa6d-5108cf2b5094/volumes"
Oct 30 21:23:47.663739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:47.663726    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf45ae6-8684-4f2d-86a2-fa2d6930f786" path="/var/lib/kubelet/pods/ebf45ae6-8684-4f2d-86a2-fa2d6930f786/volumes"
Oct 30 21:23:48.060107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.060068    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9" podUID="96a59394-b598-43d8-a666-415395af7d2e" containerName="payload-processing" containerID="cri-o://cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553" gracePeriod=25
Oct 30 21:23:48.418730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.418709    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:48.565731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.565692    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-plugins-config-volume\") pod \"96a59394-b598-43d8-a666-415395af7d2e\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") "
Oct 30 21:23:48.565731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.565733    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-kube-api-access-8fnld\" (UniqueName: \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-kube-api-access-8fnld\") pod \"96a59394-b598-43d8-a666-415395af7d2e\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") "
Oct 30 21:23:48.565997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.565786    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-routing-overlay-volume\") pod \"96a59394-b598-43d8-a666-415395af7d2e\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") "
Oct 30 21:23:48.565997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.565953    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-provider-credentials\") pod \"96a59394-b598-43d8-a666-415395af7d2e\" (UID: \"96a59394-b598-43d8-a666-415395af7d2e\") "
Oct 30 21:23:48.566214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.566187    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-plugins-config-volume" pod "96a59394-b598-43d8-a666-415395af7d2e" (UID: "96a59394-b598-43d8-a666-415395af7d2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:48.566272 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.566254    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-routing-overlay-volume" pod "96a59394-b598-43d8-a666-415395af7d2e" (UID: "96a59394-b598-43d8-a666-415395af7d2e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:48.566579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.566561    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.566652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.566580    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96a59394-b598-43d8-a666-415395af7d2e-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.568189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.568163    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-kube-api-access-8fnld" pod "96a59394-b598-43d8-a666-415395af7d2e" (UID: "96a59394-b598-43d8-a666-415395af7d2e"). InnerVolumeSpecName "kube-api-access-8fnld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:48.568189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.568179    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-provider-credentials" pod "96a59394-b598-43d8-a666-415395af7d2e" (UID: "96a59394-b598-43d8-a666-415395af7d2e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:48.593882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.592184    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:23:48.593882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.592240    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 30 21:23:48.628713 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.628678    4810 generic.go:356] "Generic (PLEG): container finished" podID="96a59394-b598-43d8-a666-415395af7d2e" containerID="cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553" exitCode=0
Oct 30 21:23:48.629305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.628897    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9" event={"ID":"96a59394-b598-43d8-a666-415395af7d2e","Type":"ContainerDied","Data":"cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553"}
Oct 30 21:23:48.629403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.628967    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"
Oct 30 21:23:48.629467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.629451    4810 scope.go:122] "RemoveContainer" containerID="cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553"
Oct 30 21:23:48.630914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.630884    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9" event={"ID":"96a59394-b598-43d8-a666-415395af7d2e","Type":"ContainerDied","Data":"1e81b58bf07a6073c7e24b4ad14ffe115af0af109108674dd9a3ecd8bd105129"}
Oct 30 21:23:48.642014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.641998    4810 scope.go:122] "RemoveContainer" containerID="cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553"
Oct 30 21:23:48.642251 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:48.642230    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553\": container with ID starting with cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553 not found: ID does not exist" containerID="cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553"
Oct 30 21:23:48.642312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.642262    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553"} err="failed to get container status \"cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553\": rpc error: code = NotFound desc = could not find container \"cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553\": container with ID starting with cffb29f71e0945edbe6dc00c8c934190a63c22394265e1143fae24589f441553 not found: ID does not exist"
Oct 30 21:23:48.657519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.657496    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"]
Oct 30 21:23:48.661371 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.661351    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1e0f5/payload-processing-external-model-e2e-extmod-b-e1e0f5-5697xqmd9"]
Oct 30 21:23:48.667672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.667648    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fnld\" (UniqueName: \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-kube-api-access-8fnld\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.667743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:48.667673    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96a59394-b598-43d8-a666-415395af7d2e-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:49.578394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.578356    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:23:49.578840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.578812    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Oct 30 21:23:49.663390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.663350    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a59394-b598-43d8-a666-415395af7d2e" path="/var/lib/kubelet/pods/96a59394-b598-43d8-a666-415395af7d2e/volumes"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.928198    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"]
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929285    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ebf45ae6-8684-4f2d-86a2-fa2d6930f786" containerName="payload-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929304    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf45ae6-8684-4f2d-86a2-fa2d6930f786" containerName="payload-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929318    4810 cpu_manager.go:414] "containerMap: removing container" podUID="96a59394-b598-43d8-a666-415395af7d2e" containerName="payload-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929323    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a59394-b598-43d8-a666-415395af7d2e" containerName="payload-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929334    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e" containerName="payload-pre-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929339    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e" containerName="payload-pre-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929373    4810 cpu_manager.go:414] "containerMap: removing container" podUID="86b2ea22-c592-4df3-aa6d-5108cf2b5094" containerName="payload-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929378    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b2ea22-c592-4df3-aa6d-5108cf2b5094" containerName="payload-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929482    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf45ae6-8684-4f2d-86a2-fa2d6930f786" containerName="payload-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929488    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a59394-b598-43d8-a666-415395af7d2e" containerName="payload-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929495    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6a9d0f-fc5a-490a-a0f1-a710e920ff2e" containerName="payload-pre-processing"
Oct 30 21:23:49.930056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.929508    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b2ea22-c592-4df3-aa6d-5108cf2b5094" containerName="payload-processing"
Oct 30 21:23:49.935797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.935463    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:49.940651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.940402    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"]
Oct 30 21:23:49.968346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:49.968316    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2ac526-openshift-default-dockercfg-rgfnz\""
Oct 30 21:23:50.025026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.024913    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-credential-socket\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.025369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.025342    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d205dba-ff07-4742-8509-44948ba1af95-istio-podinfo\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.025575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.025558    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-certs\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.025875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.025842    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-data\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.026045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.026030    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d205dba-ff07-4742-8509-44948ba1af95-istiod-ca-cert\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.026227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.026210    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-envoy\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.026361 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.026348    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-socket\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.026497 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.026484    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-istio-token\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.026701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.026686    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcbh\" (UniqueName: \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-kube-api-access-bbcbh\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.127836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.127790    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-certs\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.128085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.127978    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-data\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.128085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.128006    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d205dba-ff07-4742-8509-44948ba1af95-istiod-ca-cert\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.128085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.128068    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-envoy\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.128259 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.128090    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-socket\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.128259 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.128125    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-istio-token\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.128259 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.128231    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcbh\" (UniqueName: \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-kube-api-access-bbcbh\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.128426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.128301    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-credential-socket\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.128426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.128332    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d205dba-ff07-4742-8509-44948ba1af95-istio-podinfo\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.129515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.129484    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-socket\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.130054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.130027    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-credential-socket\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.130293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.130273    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-data\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.130553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.130530    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-certs\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.131214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.131190    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d205dba-ff07-4742-8509-44948ba1af95-istiod-ca-cert\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.164931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.152494    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d205dba-ff07-4742-8509-44948ba1af95-istio-podinfo\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.164931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.164127    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-istio-token\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.165196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.164973    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcbh\" (UniqueName: \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-kube-api-access-bbcbh\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.165268 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.165230    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-envoy\") pod \"e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") " pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.275929 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.275122    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:50.554074 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.553997    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:23:50.590054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.590018    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:50.590959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.590932    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:50.591968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.591925    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:23:50.662875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.657401    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"]
Oct 30 21:23:50.754932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.754877    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:50.755706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.755672    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:50.757151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.756022    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:50.757409 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.756612    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"]
Oct 30 21:23:50.774367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.774343    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"]
Oct 30 21:23:50.774614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.774598    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:50.788926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.788904    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ffce13-serving-cert\""
Oct 30 21:23:50.901919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.872972    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:50.905498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.902229    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a2b983-ff5b-411f-85de-01a6dbae639e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:50.905498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.903367    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46p6\" (UniqueName: \"kubernetes.io/projected/63a2b983-ff5b-411f-85de-01a6dbae639e-kube-api-access-z46p6\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:50.905498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:50.903465    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-metrics-certs\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:51.004581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.004541    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:51.004770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.004597    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a2b983-ff5b-411f-85de-01a6dbae639e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:51.004770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.004678    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46p6\" (UniqueName: \"kubernetes.io/projected/63a2b983-ff5b-411f-85de-01a6dbae639e-kube-api-access-z46p6\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:51.004770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.004717    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-metrics-certs\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:51.006201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.006161    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a2b983-ff5b-411f-85de-01a6dbae639e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:51.009393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.008741    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-metrics-certs\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:51.009393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.008845    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:51.015783 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.015399    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46p6\" (UniqueName: \"kubernetes.io/projected/63a2b983-ff5b-411f-85de-01a6dbae639e-kube-api-access-z46p6\") pod \"maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:51.170120 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.170084    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:51.539560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.539470    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"]
Oct 30 21:23:51.709138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.709049    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk" event={"ID":"63a2b983-ff5b-411f-85de-01a6dbae639e","Type":"ContainerStarted","Data":"3d5d66349b70f7793efe1ff84fd0fe8d3db01f5e132fa8dc2bfe6b3e6d9667a0"}
Oct 30 21:23:51.716976 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.716822    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" event={"ID":"9d205dba-ff07-4742-8509-44948ba1af95","Type":"ContainerStarted","Data":"07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73"}
Oct 30 21:23:51.717280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.717253    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" event={"ID":"9d205dba-ff07-4742-8509-44948ba1af95","Type":"ContainerStarted","Data":"bab619f23ab77507b866f8f9bf8e7e309546fc5fd25db1c0f03487e43a91f909"}
Oct 30 21:23:51.754370 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:51.751876    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" podStartSLOduration=2.751843251 podStartE2EDuration="2.751843251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:51.749083189 +0000 UTC m=+2798.240899735" watchObservedRunningTime="2026-10-30 21:23:51.751843251 +0000 UTC m=+2798.243659796"
Oct 30 21:23:52.276308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:52.276177    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:52.282681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:52.282652    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:52.733559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:52.733526    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk" event={"ID":"63a2b983-ff5b-411f-85de-01a6dbae639e","Type":"ContainerStarted","Data":"fbeb6975061ca04e46a841d9fb643c5dfab272896319e2c635461b0ee54856ea"}
Oct 30 21:23:52.733747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:52.733595    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:23:52.734097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:52.734077    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:52.735103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:52.735079    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:23:52.754032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:52.753994    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk" podStartSLOduration=2.323782991 podStartE2EDuration="2.753980719s" totalImagesPullingTime="430.197728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:52.751160255 +0000 UTC m=+2799.242976997" watchObservedRunningTime="2026-10-30 21:23:52.753980719 +0000 UTC m=+2799.245797300"
Oct 30 21:23:55.164161 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:23:55.164125    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:23:55.199789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.199757    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"]
Oct 30 21:23:55.206233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.206206    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:23:55.212601 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.212574    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"]
Oct 30 21:23:55.213656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.213634    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2ac526\""
Oct 30 21:23:55.226666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.226640    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2kf\" (UniqueName: \"kubernetes.io/projected/6b22d659-57de-4142-8da1-8b0705a58fc0-kube-api-access-mc2kf\") pod \"payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6\" (UID: \"6b22d659-57de-4142-8da1-8b0705a58fc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:23:55.227017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.226831    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b22d659-57de-4142-8da1-8b0705a58fc0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6\" (UID: \"6b22d659-57de-4142-8da1-8b0705a58fc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:23:55.230416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.230390    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"]
Oct 30 21:23:55.249046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.249015    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:23:55.257382 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.257356    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2ac526-dockercfg-wwjck\""
Oct 30 21:23:55.267691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.267665    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"]
Oct 30 21:23:55.329843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.328604    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2kf\" (UniqueName: \"kubernetes.io/projected/6b22d659-57de-4142-8da1-8b0705a58fc0-kube-api-access-mc2kf\") pod \"payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6\" (UID: \"6b22d659-57de-4142-8da1-8b0705a58fc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:23:55.329843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.328817    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b22d659-57de-4142-8da1-8b0705a58fc0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6\" (UID: \"6b22d659-57de-4142-8da1-8b0705a58fc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:23:55.329843 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.329787    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b22d659-57de-4142-8da1-8b0705a58fc0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6\" (UID: \"6b22d659-57de-4142-8da1-8b0705a58fc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:23:55.393884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.390688    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2kf\" (UniqueName: \"kubernetes.io/projected/6b22d659-57de-4142-8da1-8b0705a58fc0-kube-api-access-mc2kf\") pod \"payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6\" (UID: \"6b22d659-57de-4142-8da1-8b0705a58fc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:23:55.433378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.433290    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5wh2\" (UniqueName: \"kubernetes.io/projected/5b2866f7-5408-4815-b3b3-c898ce73181f-kube-api-access-v5wh2\") pod \"payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8\" (UID: \"5b2866f7-5408-4815-b3b3-c898ce73181f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:23:55.433650 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.433626    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2866f7-5408-4815-b3b3-c898ce73181f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8\" (UID: \"5b2866f7-5408-4815-b3b3-c898ce73181f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:23:55.534981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.534945    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5wh2\" (UniqueName: \"kubernetes.io/projected/5b2866f7-5408-4815-b3b3-c898ce73181f-kube-api-access-v5wh2\") pod \"payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8\" (UID: \"5b2866f7-5408-4815-b3b3-c898ce73181f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:23:55.535187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.534997    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2866f7-5408-4815-b3b3-c898ce73181f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8\" (UID: \"5b2866f7-5408-4815-b3b3-c898ce73181f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:23:55.536007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.535984    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2866f7-5408-4815-b3b3-c898ce73181f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8\" (UID: \"5b2866f7-5408-4815-b3b3-c898ce73181f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:23:55.543640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.543616    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5wh2\" (UniqueName: \"kubernetes.io/projected/5b2866f7-5408-4815-b3b3-c898ce73181f-kube-api-access-v5wh2\") pod \"payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8\" (UID: \"5b2866f7-5408-4815-b3b3-c898ce73181f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:23:55.552445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.552420    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:23:55.582602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.582578    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:23:55.874152 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.874116    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"]
Oct 30 21:23:55.875299 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:55.875267    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b22d659_57de_4142_8da1_8b0705a58fc0.slice/crio-75aef960c55f1a1cbee0eccb1ad0e5704f0fb4e58193a5d950d02fde0a6b7c96 WatchSource:0}: Error finding container 75aef960c55f1a1cbee0eccb1ad0e5704f0fb4e58193a5d950d02fde0a6b7c96: Status 404 returned error can't find the container with id 75aef960c55f1a1cbee0eccb1ad0e5704f0fb4e58193a5d950d02fde0a6b7c96
Oct 30 21:23:55.901789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:55.901765    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"]
Oct 30 21:23:55.903141 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:23:55.903115    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2866f7_5408_4815_b3b3_c898ce73181f.slice/crio-6771d717ab17d272daff4413344a316f01237475857a54d972ad5e076622a90b WatchSource:0}: Error finding container 6771d717ab17d272daff4413344a316f01237475857a54d972ad5e076622a90b: Status 404 returned error can't find the container with id 6771d717ab17d272daff4413344a316f01237475857a54d972ad5e076622a90b
Oct 30 21:23:56.774693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:56.774656    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8" event={"ID":"5b2866f7-5408-4815-b3b3-c898ce73181f","Type":"ContainerStarted","Data":"b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4"}
Oct 30 21:23:56.775209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:56.775186    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:23:56.775343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:56.775219    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8" event={"ID":"5b2866f7-5408-4815-b3b3-c898ce73181f","Type":"ContainerStarted","Data":"6771d717ab17d272daff4413344a316f01237475857a54d972ad5e076622a90b"}
Oct 30 21:23:56.776359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:56.776329    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6" event={"ID":"6b22d659-57de-4142-8da1-8b0705a58fc0","Type":"ContainerStarted","Data":"b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc"}
Oct 30 21:23:56.776467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:56.776378    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:23:56.776467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:56.776411    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6" event={"ID":"6b22d659-57de-4142-8da1-8b0705a58fc0","Type":"ContainerStarted","Data":"75aef960c55f1a1cbee0eccb1ad0e5704f0fb4e58193a5d950d02fde0a6b7c96"}
Oct 30 21:23:56.796451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:56.796401    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8" podStartSLOduration=1.7963893450000001 podStartE2EDuration="1.796389345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:56.789746179 +0000 UTC m=+2803.281562725" watchObservedRunningTime="2026-10-30 21:23:56.796389345 +0000 UTC m=+2803.288205890"
Oct 30 21:23:56.805263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:56.805207    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6" podStartSLOduration=1.80519093 podStartE2EDuration="1.80519093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:56.804711925 +0000 UTC m=+2803.296528481" watchObservedRunningTime="2026-10-30 21:23:56.80519093 +0000 UTC m=+2803.297007476"
Oct 30 21:23:58.748291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:23:58.748262    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:24:02.720549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:02.720510    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"]
Oct 30 21:24:02.725314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:02.725285    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-345bc4-dgzvt"]
Oct 30 21:24:03.677704 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:03.677667    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf757219-2452-43fb-9fa9-eeafc55695b6" path="/var/lib/kubelet/pods/cf757219-2452-43fb-9fa9-eeafc55695b6/volumes"
Oct 30 21:24:07.790522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:07.790493    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:24:08.848405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:08.848366    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:24:10.409376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:10.409338    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"]
Oct 30 21:24:10.410195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:10.410161    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerName="istio-proxy" containerID="cri-o://0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6" gracePeriod=30
Oct 30 21:24:10.875648 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:10.875562    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"]
Oct 30 21:24:10.881998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:10.881971    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:10.888798 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:10.888765    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"]
Oct 30 21:24:10.888965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:10.888946    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:10.889185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:10.889164    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:24:11.022482 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.022447    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:11.022677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.022539    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:11.022792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.022760    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nbft\" (UniqueName: \"kubernetes.io/projected/401aac4b-3f89-4450-9cb8-e831f4a68352-kube-api-access-8nbft\") pod \"maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:11.124596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.124563    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:11.124763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.124643    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:11.124808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.124767    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nbft\" (UniqueName: \"kubernetes.io/projected/401aac4b-3f89-4450-9cb8-e831f4a68352-kube-api-access-8nbft\") pod \"maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:11.125440 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.125414    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:11.127117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.127058    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:11.133635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.133606    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nbft\" (UniqueName: \"kubernetes.io/projected/401aac4b-3f89-4450-9cb8-e831f4a68352-kube-api-access-8nbft\") pod \"maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:11.204015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.203920    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:11.508397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:11.508370    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"]
Oct 30 21:24:11.508935 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:24:11.508911    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401aac4b_3f89_4450_9cb8_e831f4a68352.slice/crio-7f6571026b843ed0f652b3a4670eb3b72c7e06e14aeab6a1ca5d1ac16a74ff43 WatchSource:0}: Error finding container 7f6571026b843ed0f652b3a4670eb3b72c7e06e14aeab6a1ca5d1ac16a74ff43: Status 404 returned error can't find the container with id 7f6571026b843ed0f652b3a4670eb3b72c7e06e14aeab6a1ca5d1ac16a74ff43
Oct 30 21:24:12.043544 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:12.043373    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv" event={"ID":"401aac4b-3f89-4450-9cb8-e831f4a68352","Type":"ContainerStarted","Data":"57d2f5ab6fcf01abd9019d7262abe657aa0c87321d8eb07952431a9f3ff191ad"}
Oct 30 21:24:12.043544 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:12.043444    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv" event={"ID":"401aac4b-3f89-4450-9cb8-e831f4a68352","Type":"ContainerStarted","Data":"7f6571026b843ed0f652b3a4670eb3b72c7e06e14aeab6a1ca5d1ac16a74ff43"}
Oct 30 21:24:12.063012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:12.062955    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv" podStartSLOduration=2.062929634 podStartE2EDuration="2.062929634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:12.059624237 +0000 UTC m=+2818.551440822" watchObservedRunningTime="2026-10-30 21:24:12.062929634 +0000 UTC m=+2818.554746180"
Oct 30 21:24:13.053189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:13.053155    4810 generic.go:356] "Generic (PLEG): container finished" podID="401aac4b-3f89-4450-9cb8-e831f4a68352" containerID="57d2f5ab6fcf01abd9019d7262abe657aa0c87321d8eb07952431a9f3ff191ad" exitCode=0
Oct 30 21:24:13.053599 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:13.053241    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv" event={"ID":"401aac4b-3f89-4450-9cb8-e831f4a68352","Type":"ContainerDied","Data":"57d2f5ab6fcf01abd9019d7262abe657aa0c87321d8eb07952431a9f3ff191ad"}
Oct 30 21:24:13.824799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:13.822438    4810 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:24:13.824799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:13.822506    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:24:14.303987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.303959    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:14.394613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.394524    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-openshift-ca\") pod \"401aac4b-3f89-4450-9cb8-e831f4a68352\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") "
Oct 30 21:24:14.394613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.394572    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401aac4b-3f89-4450-9cb8-e831f4a68352-kube-api-access-8nbft\" (UniqueName: \"kubernetes.io/projected/401aac4b-3f89-4450-9cb8-e831f4a68352-kube-api-access-8nbft\") pod \"401aac4b-3f89-4450-9cb8-e831f4a68352\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") "
Oct 30 21:24:14.394850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.394772    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-xks-ca\") pod \"401aac4b-3f89-4450-9cb8-e831f4a68352\" (UID: \"401aac4b-3f89-4450-9cb8-e831f4a68352\") "
Oct 30 21:24:14.394996 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.394966    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-openshift-ca" pod "401aac4b-3f89-4450-9cb8-e831f4a68352" (UID: "401aac4b-3f89-4450-9cb8-e831f4a68352"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:14.395393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.395369    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:14.397396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.397367    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-xks-ca" pod "401aac4b-3f89-4450-9cb8-e831f4a68352" (UID: "401aac4b-3f89-4450-9cb8-e831f4a68352"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:14.397540 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.397515    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401aac4b-3f89-4450-9cb8-e831f4a68352-kube-api-access-8nbft" pod "401aac4b-3f89-4450-9cb8-e831f4a68352" (UID: "401aac4b-3f89-4450-9cb8-e831f4a68352"). InnerVolumeSpecName "kube-api-access-8nbft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:14.496553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.496512    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401aac4b-3f89-4450-9cb8-e831f4a68352-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:14.496553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.496550    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nbft\" (UniqueName: \"kubernetes.io/projected/401aac4b-3f89-4450-9cb8-e831f4a68352-kube-api-access-8nbft\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:14.777028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.776950    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"]
Oct 30 21:24:14.778243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.778222    4810 cpu_manager.go:414] "containerMap: removing container" podUID="401aac4b-3f89-4450-9cb8-e831f4a68352" containerName="revoke-keys"
Oct 30 21:24:14.778305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.778246    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401aac4b-3f89-4450-9cb8-e831f4a68352" containerName="revoke-keys"
Oct 30 21:24:14.778459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.778448    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="401aac4b-3f89-4450-9cb8-e831f4a68352" containerName="revoke-keys"
Oct 30 21:24:14.782992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.782965    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:24:14.787161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.787130    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5506f4-dockercfg-pfdgh\""
Oct 30 21:24:14.787271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.787212    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5506f4\""
Oct 30 21:24:14.792637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.792612    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"]
Oct 30 21:24:14.819543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.819514    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"]
Oct 30 21:24:14.825780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.825759    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:24:14.837990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.837963    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"]
Oct 30 21:24:14.908879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.906241    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wfd5\" (UniqueName: \"kubernetes.io/projected/a6c727b1-6fe8-46eb-95ff-4748528ee68d-kube-api-access-5wfd5\") pod \"payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2\" (UID: \"a6c727b1-6fe8-46eb-95ff-4748528ee68d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:24:14.908879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.906291    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jrl\" (UniqueName: \"kubernetes.io/projected/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-kube-api-access-47jrl\") pod \"payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj\" (UID: \"f268e880-2e55-4ff3-837d-a4fb2abbf8a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:24:14.908879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.906326    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj\" (UID: \"f268e880-2e55-4ff3-837d-a4fb2abbf8a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:24:14.908879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:14.906490    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c727b1-6fe8-46eb-95ff-4748528ee68d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2\" (UID: \"a6c727b1-6fe8-46eb-95ff-4748528ee68d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:24:15.008889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.007946    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c727b1-6fe8-46eb-95ff-4748528ee68d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2\" (UID: \"a6c727b1-6fe8-46eb-95ff-4748528ee68d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:24:15.008889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.008134    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfd5\" (UniqueName: \"kubernetes.io/projected/a6c727b1-6fe8-46eb-95ff-4748528ee68d-kube-api-access-5wfd5\") pod \"payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2\" (UID: \"a6c727b1-6fe8-46eb-95ff-4748528ee68d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:24:15.008889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.008162    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jrl\" (UniqueName: \"kubernetes.io/projected/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-kube-api-access-47jrl\") pod \"payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj\" (UID: \"f268e880-2e55-4ff3-837d-a4fb2abbf8a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:24:15.008889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.008199    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj\" (UID: \"f268e880-2e55-4ff3-837d-a4fb2abbf8a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:24:15.009267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.008973    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c727b1-6fe8-46eb-95ff-4748528ee68d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2\" (UID: \"a6c727b1-6fe8-46eb-95ff-4748528ee68d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:24:15.009332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.009288    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj\" (UID: \"f268e880-2e55-4ff3-837d-a4fb2abbf8a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:24:15.028824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.022637    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wfd5\" (UniqueName: \"kubernetes.io/projected/a6c727b1-6fe8-46eb-95ff-4748528ee68d-kube-api-access-5wfd5\") pod \"payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2\" (UID: \"a6c727b1-6fe8-46eb-95ff-4748528ee68d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:24:15.028824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.028146    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jrl\" (UniqueName: \"kubernetes.io/projected/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-kube-api-access-47jrl\") pod \"payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj\" (UID: \"f268e880-2e55-4ff3-837d-a4fb2abbf8a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:24:15.072525 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.072489    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"
Oct 30 21:24:15.072678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.072522    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv" event={"ID":"401aac4b-3f89-4450-9cb8-e831f4a68352","Type":"ContainerDied","Data":"7f6571026b843ed0f652b3a4670eb3b72c7e06e14aeab6a1ca5d1ac16a74ff43"}
Oct 30 21:24:15.072678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.072575    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f6571026b843ed0f652b3a4670eb3b72c7e06e14aeab6a1ca5d1ac16a74ff43"
Oct 30 21:24:15.108449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.108421    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:24:15.147159 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.147126    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:24:15.426312 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:24:15.426262    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c727b1_6fe8_46eb_95ff_4748528ee68d.slice/crio-bd394428beaff7385d2c7c78aeac6a6c36ab48f00343ceb0d1fe6aa0d3363e67 WatchSource:0}: Error finding container bd394428beaff7385d2c7c78aeac6a6c36ab48f00343ceb0d1fe6aa0d3363e67: Status 404 returned error can't find the container with id bd394428beaff7385d2c7c78aeac6a6c36ab48f00343ceb0d1fe6aa0d3363e67
Oct 30 21:24:15.426706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.426682    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"]
Oct 30 21:24:15.491844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.478902    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"]
Oct 30 21:24:15.534432 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:24:15.534349    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf268e880_2e55_4ff3_837d_a4fb2abbf8a1.slice/crio-ba050f29b95b10f82d1d8ea53aa4ee1fffaba1860b8123624a12ee9c41e94436 WatchSource:0}: Error finding container ba050f29b95b10f82d1d8ea53aa4ee1fffaba1860b8123624a12ee9c41e94436: Status 404 returned error can't find the container with id ba050f29b95b10f82d1d8ea53aa4ee1fffaba1860b8123624a12ee9c41e94436
Oct 30 21:24:15.713989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.712976    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:24:15.821565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.821457    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-podinfo\") pod \"f6919133-5577-44c1-bdb0-565eb839eb6e\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") "
Oct 30 21:24:15.821812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.821791    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-credential-socket\") pod \"f6919133-5577-44c1-bdb0-565eb839eb6e\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") "
Oct 30 21:24:15.821974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.821958    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-kube-api-access-ghcpb\" (UniqueName: \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-kube-api-access-ghcpb\") pod \"f6919133-5577-44c1-bdb0-565eb839eb6e\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") "
Oct 30 21:24:15.822164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.822130    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-certs\") pod \"f6919133-5577-44c1-bdb0-565eb839eb6e\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") "
Oct 30 21:24:15.822403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.822376    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-data\") pod \"f6919133-5577-44c1-bdb0-565eb839eb6e\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") "
Oct 30 21:24:15.822546 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.822441    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-envoy\") pod \"f6919133-5577-44c1-bdb0-565eb839eb6e\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") "
Oct 30 21:24:15.822546 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.822472    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6919133-5577-44c1-bdb0-565eb839eb6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6919133-5577-44c1-bdb0-565eb839eb6e-istiod-ca-cert\") pod \"f6919133-5577-44c1-bdb0-565eb839eb6e\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") "
Oct 30 21:24:15.822546 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.822522    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-token\" (UniqueName: \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-token\") pod \"f6919133-5577-44c1-bdb0-565eb839eb6e\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") "
Oct 30 21:24:15.822733 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.822635    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-socket\") pod \"f6919133-5577-44c1-bdb0-565eb839eb6e\" (UID: \"f6919133-5577-44c1-bdb0-565eb839eb6e\") "
Oct 30 21:24:15.824283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.824081    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-credential-socket" pod "f6919133-5577-44c1-bdb0-565eb839eb6e" (UID: "f6919133-5577-44c1-bdb0-565eb839eb6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:15.824535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.824512    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-certs" pod "f6919133-5577-44c1-bdb0-565eb839eb6e" (UID: "f6919133-5577-44c1-bdb0-565eb839eb6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:15.824712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.824691    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-socket" pod "f6919133-5577-44c1-bdb0-565eb839eb6e" (UID: "f6919133-5577-44c1-bdb0-565eb839eb6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:15.825060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.824992    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-data" pod "f6919133-5577-44c1-bdb0-565eb839eb6e" (UID: "f6919133-5577-44c1-bdb0-565eb839eb6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:15.825173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.825117    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6919133-5577-44c1-bdb0-565eb839eb6e-istiod-ca-cert" pod "f6919133-5577-44c1-bdb0-565eb839eb6e" (UID: "f6919133-5577-44c1-bdb0-565eb839eb6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:15.827605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.827577    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-token" pod "f6919133-5577-44c1-bdb0-565eb839eb6e" (UID: "f6919133-5577-44c1-bdb0-565eb839eb6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:15.830314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.830226    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-kube-api-access-ghcpb" pod "f6919133-5577-44c1-bdb0-565eb839eb6e" (UID: "f6919133-5577-44c1-bdb0-565eb839eb6e"). InnerVolumeSpecName "kube-api-access-ghcpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:15.830314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.830290    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-podinfo" pod "f6919133-5577-44c1-bdb0-565eb839eb6e" (UID: "f6919133-5577-44c1-bdb0-565eb839eb6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:24:15.831826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.831806    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-envoy" pod "f6919133-5577-44c1-bdb0-565eb839eb6e" (UID: "f6919133-5577-44c1-bdb0-565eb839eb6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:15.924728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.924694    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.924728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.924722    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.924728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.924732    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghcpb\" (UniqueName: \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-kube-api-access-ghcpb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.925023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.924741    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.925023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.924751    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.925023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.924769    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.925023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.924778    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6919133-5577-44c1-bdb0-565eb839eb6e-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.925023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.924789    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6919133-5577-44c1-bdb0-565eb839eb6e-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.925023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:15.924797    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6919133-5577-44c1-bdb0-565eb839eb6e-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:16.086394 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.086304    4810 generic.go:356] "Generic (PLEG): container finished" podID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerID="0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6" exitCode=0
Oct 30 21:24:16.086573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.086400    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"
Oct 30 21:24:16.086573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.086431    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" event={"ID":"f6919133-5577-44c1-bdb0-565eb839eb6e","Type":"ContainerDied","Data":"0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6"}
Oct 30 21:24:16.086573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.086489    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc" event={"ID":"f6919133-5577-44c1-bdb0-565eb839eb6e","Type":"ContainerDied","Data":"dd0e438ba327e055437d878e358b2b3f109d3094e55b9521f5bb8c5509758470"}
Oct 30 21:24:16.086573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.086518    4810 scope.go:122] "RemoveContainer" containerID="0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6"
Oct 30 21:24:16.088819 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.088790    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2" event={"ID":"a6c727b1-6fe8-46eb-95ff-4748528ee68d","Type":"ContainerStarted","Data":"128afce246826cbda2b3a0ec5b0a922a8f746654ede8d6869d97174b578e0049"}
Oct 30 21:24:16.089010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.088844    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2" event={"ID":"a6c727b1-6fe8-46eb-95ff-4748528ee68d","Type":"ContainerStarted","Data":"bd394428beaff7385d2c7c78aeac6a6c36ab48f00343ceb0d1fe6aa0d3363e67"}
Oct 30 21:24:16.089010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.088989    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:24:16.090828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.090801    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj" event={"ID":"f268e880-2e55-4ff3-837d-a4fb2abbf8a1","Type":"ContainerStarted","Data":"d9d0439b665373fee670ec399653e8c1d39db1b98ad5a30a3a2548ece0f5f256"}
Oct 30 21:24:16.090998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.090852    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj" event={"ID":"f268e880-2e55-4ff3-837d-a4fb2abbf8a1","Type":"ContainerStarted","Data":"ba050f29b95b10f82d1d8ea53aa4ee1fffaba1860b8123624a12ee9c41e94436"}
Oct 30 21:24:16.091072 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.091053    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:24:16.102724 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.102701    4810 scope.go:122] "RemoveContainer" containerID="0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6"
Oct 30 21:24:16.104020 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:24:16.103993    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6\": container with ID starting with 0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6 not found: ID does not exist" containerID="0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6"
Oct 30 21:24:16.104111 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.104029    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6"} err="failed to get container status \"0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6\": rpc error: code = NotFound desc = could not find container \"0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6\": container with ID starting with 0b1896c8ad840adce36251abefc858b243613b383b5b23c15febb5528c6250b6 not found: ID does not exist"
Oct 30 21:24:16.112619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.112570    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2" podStartSLOduration=2.112553674 podStartE2EDuration="2.112553674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:16.106363866 +0000 UTC m=+2822.598180411" watchObservedRunningTime="2026-10-30 21:24:16.112553674 +0000 UTC m=+2822.604370220"
Oct 30 21:24:16.133017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.132968    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj" podStartSLOduration=2.132952484 podStartE2EDuration="2.132952484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:16.124913782 +0000 UTC m=+2822.616730329" watchObservedRunningTime="2026-10-30 21:24:16.132952484 +0000 UTC m=+2822.624769029"
Oct 30 21:24:16.146207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.146158    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"]
Oct 30 21:24:16.150642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:16.150615    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-hdsjc"]
Oct 30 21:24:17.665173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:17.665143    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" path="/var/lib/kubelet/pods/f6919133-5577-44c1-bdb0-565eb839eb6e/volumes"
Oct 30 21:24:20.405879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:20.405342    4810 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d5dmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:24:20.405879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:20.405410    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d5dmn" podUID="f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:21.068880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:21.067918    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"]
Oct 30 21:24:21.329945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:21.328018    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:24.030302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:24.030153    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"]
Oct 30 21:24:24.035774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:24.035735    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d973926a-s7mbx"]
Oct 30 21:24:25.157305 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:24:25.157223    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:24:25.664767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:25.664733    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a321c97-7975-4980-b689-df29c5f9e0c1" path="/var/lib/kubelet/pods/8a321c97-7975-4980-b689-df29c5f9e0c1/volumes"
Oct 30 21:24:26.068390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:26.068269    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn" podUID="5530ac61-52fa-400c-9506-5a7ea1edb7db" containerName="payload-pre-processing" containerID="cri-o://8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604" gracePeriod=25
Oct 30 21:24:27.109573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.109535    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:24:27.111388 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.111365    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:24:27.288743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.288699    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"]
Oct 30 21:24:27.290005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.289978    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerName="istio-proxy"
Oct 30 21:24:27.290005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.290001    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerName="istio-proxy"
Oct 30 21:24:27.290280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.290266    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6919133-5577-44c1-bdb0-565eb839eb6e" containerName="istio-proxy"
Oct 30 21:24:27.297596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.297568    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:24:27.298042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.298017    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"]
Oct 30 21:24:27.306931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.306904    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9b78e4\""
Oct 30 21:24:27.307256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.307235    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:24:27.308889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.308834    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9b78e4-dockercfg-lhcpm\""
Oct 30 21:24:27.309969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.309287    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"]
Oct 30 21:24:27.348007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.347634    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"]
Oct 30 21:24:27.366632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.366459    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a804f13-7cd2-4949-bd3d-ecf594465e0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d\" (UID: \"3a804f13-7cd2-4949-bd3d-ecf594465e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:24:27.366632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.366534    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d8eef0-e5c5-4f87-9502-973ef001a54e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv\" (UID: \"96d8eef0-e5c5-4f87-9502-973ef001a54e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:24:27.366801 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.366633    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzn2\" (UniqueName: \"kubernetes.io/projected/96d8eef0-e5c5-4f87-9502-973ef001a54e-kube-api-access-nlzn2\") pod \"payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv\" (UID: \"96d8eef0-e5c5-4f87-9502-973ef001a54e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:24:27.366969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.366943    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddr57\" (UniqueName: \"kubernetes.io/projected/3a804f13-7cd2-4949-bd3d-ecf594465e0b-kube-api-access-ddr57\") pod \"payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d\" (UID: \"3a804f13-7cd2-4949-bd3d-ecf594465e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:24:27.484881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.481607    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d8eef0-e5c5-4f87-9502-973ef001a54e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv\" (UID: \"96d8eef0-e5c5-4f87-9502-973ef001a54e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:24:27.484881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.481671    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzn2\" (UniqueName: \"kubernetes.io/projected/96d8eef0-e5c5-4f87-9502-973ef001a54e-kube-api-access-nlzn2\") pod \"payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv\" (UID: \"96d8eef0-e5c5-4f87-9502-973ef001a54e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:24:27.484881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.481909    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddr57\" (UniqueName: \"kubernetes.io/projected/3a804f13-7cd2-4949-bd3d-ecf594465e0b-kube-api-access-ddr57\") pod \"payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d\" (UID: \"3a804f13-7cd2-4949-bd3d-ecf594465e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:24:27.484881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.481992    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a804f13-7cd2-4949-bd3d-ecf594465e0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d\" (UID: \"3a804f13-7cd2-4949-bd3d-ecf594465e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:24:27.508650 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.508448    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d8eef0-e5c5-4f87-9502-973ef001a54e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv\" (UID: \"96d8eef0-e5c5-4f87-9502-973ef001a54e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:24:27.508832 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.508675    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a804f13-7cd2-4949-bd3d-ecf594465e0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d\" (UID: \"3a804f13-7cd2-4949-bd3d-ecf594465e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:24:27.518878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.515743    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzn2\" (UniqueName: \"kubernetes.io/projected/96d8eef0-e5c5-4f87-9502-973ef001a54e-kube-api-access-nlzn2\") pod \"payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv\" (UID: \"96d8eef0-e5c5-4f87-9502-973ef001a54e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:24:27.538036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.537918    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddr57\" (UniqueName: \"kubernetes.io/projected/3a804f13-7cd2-4949-bd3d-ecf594465e0b-kube-api-access-ddr57\") pod \"payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d\" (UID: \"3a804f13-7cd2-4949-bd3d-ecf594465e0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:24:27.649460 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.649396    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:24:27.652279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:27.652248    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:24:28.035208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.035180    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"]
Oct 30 21:24:28.035351 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:24:28.035178    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d8eef0_e5c5_4f87_9502_973ef001a54e.slice/crio-acdbdf5579639d12322895fda1ad2498a1785cca140edeb001795f67847d5e73 WatchSource:0}: Error finding container acdbdf5579639d12322895fda1ad2498a1785cca140edeb001795f67847d5e73: Status 404 returned error can't find the container with id acdbdf5579639d12322895fda1ad2498a1785cca140edeb001795f67847d5e73
Oct 30 21:24:28.041339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.041312    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"]
Oct 30 21:24:28.043123 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:24:28.043099    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a804f13_7cd2_4949_bd3d_ecf594465e0b.slice/crio-bd5acda579acf1ef1eb3185ccf868199ad139ed0e3d4ceceecd0658626a8215a WatchSource:0}: Error finding container bd5acda579acf1ef1eb3185ccf868199ad139ed0e3d4ceceecd0658626a8215a: Status 404 returned error can't find the container with id bd5acda579acf1ef1eb3185ccf868199ad139ed0e3d4ceceecd0658626a8215a
Oct 30 21:24:28.339614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.339536    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn" podUID="5530ac61-52fa-400c-9506-5a7ea1edb7db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:28.752086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.752042    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv" event={"ID":"96d8eef0-e5c5-4f87-9502-973ef001a54e","Type":"ContainerStarted","Data":"6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323"}
Oct 30 21:24:28.752302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.752112    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv" event={"ID":"96d8eef0-e5c5-4f87-9502-973ef001a54e","Type":"ContainerStarted","Data":"acdbdf5579639d12322895fda1ad2498a1785cca140edeb001795f67847d5e73"}
Oct 30 21:24:28.752302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.752143    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:24:28.753927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.753897    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d" event={"ID":"3a804f13-7cd2-4949-bd3d-ecf594465e0b","Type":"ContainerStarted","Data":"f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934"}
Oct 30 21:24:28.754043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.753943    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d" event={"ID":"3a804f13-7cd2-4949-bd3d-ecf594465e0b","Type":"ContainerStarted","Data":"bd5acda579acf1ef1eb3185ccf868199ad139ed0e3d4ceceecd0658626a8215a"}
Oct 30 21:24:28.754043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.753967    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:24:28.772782 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.772737    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv" podStartSLOduration=1.772724428 podStartE2EDuration="1.772724428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:28.769130654 +0000 UTC m=+2835.260947197" watchObservedRunningTime="2026-10-30 21:24:28.772724428 +0000 UTC m=+2835.264540973"
Oct 30 21:24:28.788178 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:28.788130    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d" podStartSLOduration=1.788113216 podStartE2EDuration="1.788113216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:28.786141564 +0000 UTC m=+2835.277958108" watchObservedRunningTime="2026-10-30 21:24:28.788113216 +0000 UTC m=+2835.279929763"
Oct 30 21:24:30.664701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:30.664662    4810 patch_prober.go:29] interesting pod/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Oct 30 21:24:30.665248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:30.664721    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Oct 30 21:24:30.784023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:30.783977    4810 generic.go:356] "Generic (PLEG): container finished" podID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerID="378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732" exitCode=0
Oct 30 21:24:30.784226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:30.784135    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" event={"ID":"ec9fbac4-7de9-488e-8181-6b9be2b154dd","Type":"ContainerDied","Data":"378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732"}
Oct 30 21:24:30.784612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:30.784589    4810 scope.go:122] "RemoveContainer" containerID="378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732"
Oct 30 21:24:30.786531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:30.786498    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:30.786636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:30.786591    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:30.786709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:30.786640    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:31.816282 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:31.816247    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" event={"ID":"ec9fbac4-7de9-488e-8181-6b9be2b154dd","Type":"ContainerStarted","Data":"ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d"}
Oct 30 21:24:32.603268 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:32.603110    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:24:32.603638 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:32.603607    4810 patch_prober.go:29] interesting pod/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Oct 30 21:24:32.603768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:32.603680    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Oct 30 21:24:33.602334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:33.602289    4810 patch_prober.go:29] interesting pod/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Oct 30 21:24:33.602836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:33.602363    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Oct 30 21:24:34.603307 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:34.603268    4810 patch_prober.go:29] interesting pod/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Oct 30 21:24:34.603810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:34.603338    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Oct 30 21:24:35.030260 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:35.030225    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"]
Oct 30 21:24:35.034537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:35.034508    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef9cc37-bk4l8"]
Oct 30 21:24:35.604284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:35.604244    4810 patch_prober.go:29] interesting pod/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Oct 30 21:24:35.605117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:35.605076    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Oct 30 21:24:35.682073 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:35.682035    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f38a2d5-c514-4d0d-b714-5149065e6e4b" path="/var/lib/kubelet/pods/1f38a2d5-c514-4d0d-b714-5149065e6e4b/volumes"
Oct 30 21:24:36.605346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:36.605310    4810 patch_prober.go:29] interesting pod/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body=
Oct 30 21:24:36.606002 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:36.605957    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused"
Oct 30 21:24:36.751947 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:36.751911    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"]
Oct 30 21:24:36.769239 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:36.769203    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a574c2-54qmv"]
Oct 30 21:24:37.626840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:37.626807    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:24:37.629052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:37.629029    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:24:37.635017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:37.633076    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:24:37.679078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:37.679051    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="401aac4b-3f89-4450-9cb8-e831f4a68352" path="/var/lib/kubelet/pods/401aac4b-3f89-4450-9cb8-e831f4a68352/volumes"
Oct 30 21:24:38.340248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:38.340208    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn" podUID="5530ac61-52fa-400c-9506-5a7ea1edb7db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:39.767434 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:39.767405    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:24:39.767872 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:39.767531    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:24:44.748625 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:44.748595    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"]
Oct 30 21:24:44.753575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:44.753556    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:44.759940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:44.759846    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:24:44.759940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:44.759923    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:44.760731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:44.760710    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"]
Oct 30 21:24:44.904183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:44.904151    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:44.904395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:44.904222    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:44.904395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:44.904305    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rggm9\" (UniqueName: \"kubernetes.io/projected/29b337e7-8252-4971-a190-417b0e4d9bca-kube-api-access-rggm9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:45.011937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:45.011389    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:45.011937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:45.011537    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:45.011937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:45.011585    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rggm9\" (UniqueName: \"kubernetes.io/projected/29b337e7-8252-4971-a190-417b0e4d9bca-kube-api-access-rggm9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:45.027945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:45.027848    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:45.031433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:45.031396    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:45.032263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:45.032229    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rggm9\" (UniqueName: \"kubernetes.io/projected/29b337e7-8252-4971-a190-417b0e4d9bca-kube-api-access-rggm9\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:45.072872 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:45.072834    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:45.372116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:45.372085    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"]
Oct 30 21:24:45.372533 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:24:45.372505    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b337e7_8252_4971_a190_417b0e4d9bca.slice/crio-31ecf6a82d649539e028e7bb81fa0dd3bc2e212002f76bdb70375039a47cde67 WatchSource:0}: Error finding container 31ecf6a82d649539e028e7bb81fa0dd3bc2e212002f76bdb70375039a47cde67: Status 404 returned error can't find the container with id 31ecf6a82d649539e028e7bb81fa0dd3bc2e212002f76bdb70375039a47cde67
Oct 30 21:24:46.070932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:46.070159    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8" event={"ID":"29b337e7-8252-4971-a190-417b0e4d9bca","Type":"ContainerStarted","Data":"cc2fabf765d0c606dd6432aa9dbe20d52f8bc39bc817b3594bf80cae9e9e7446"}
Oct 30 21:24:46.070932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:46.070233    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8" event={"ID":"29b337e7-8252-4971-a190-417b0e4d9bca","Type":"ContainerStarted","Data":"31ecf6a82d649539e028e7bb81fa0dd3bc2e212002f76bdb70375039a47cde67"}
Oct 30 21:24:46.101920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:46.100707    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8" podStartSLOduration=2.100686721 podStartE2EDuration="2.100686721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:46.093546129 +0000 UTC m=+2852.585362665" watchObservedRunningTime="2026-10-30 21:24:46.100686721 +0000 UTC m=+2852.592503270"
Oct 30 21:24:46.424371 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:46.424343    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:24:46.536339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:46.536313    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5530ac61-52fa-400c-9506-5a7ea1edb7db-kube-api-access-bnzvc\" (UniqueName: \"kubernetes.io/projected/5530ac61-52fa-400c-9506-5a7ea1edb7db-kube-api-access-bnzvc\") pod \"5530ac61-52fa-400c-9506-5a7ea1edb7db\" (UID: \"5530ac61-52fa-400c-9506-5a7ea1edb7db\") "
Oct 30 21:24:46.536486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:46.536433    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5530ac61-52fa-400c-9506-5a7ea1edb7db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5530ac61-52fa-400c-9506-5a7ea1edb7db-plugins-config-volume\") pod \"5530ac61-52fa-400c-9506-5a7ea1edb7db\" (UID: \"5530ac61-52fa-400c-9506-5a7ea1edb7db\") "
Oct 30 21:24:46.536801 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:46.536774    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5530ac61-52fa-400c-9506-5a7ea1edb7db-plugins-config-volume" pod "5530ac61-52fa-400c-9506-5a7ea1edb7db" (UID: "5530ac61-52fa-400c-9506-5a7ea1edb7db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:46.538304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:46.538282    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5530ac61-52fa-400c-9506-5a7ea1edb7db-kube-api-access-bnzvc" pod "5530ac61-52fa-400c-9506-5a7ea1edb7db" (UID: "5530ac61-52fa-400c-9506-5a7ea1edb7db"). InnerVolumeSpecName "kube-api-access-bnzvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:46.638352 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:46.638319    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5530ac61-52fa-400c-9506-5a7ea1edb7db-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.638498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:46.638356    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnzvc\" (UniqueName: \"kubernetes.io/projected/5530ac61-52fa-400c-9506-5a7ea1edb7db-kube-api-access-bnzvc\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:47.082775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.082732    4810 generic.go:356] "Generic (PLEG): container finished" podID="29b337e7-8252-4971-a190-417b0e4d9bca" containerID="cc2fabf765d0c606dd6432aa9dbe20d52f8bc39bc817b3594bf80cae9e9e7446" exitCode=0
Oct 30 21:24:47.083233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.082784    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8" event={"ID":"29b337e7-8252-4971-a190-417b0e4d9bca","Type":"ContainerDied","Data":"cc2fabf765d0c606dd6432aa9dbe20d52f8bc39bc817b3594bf80cae9e9e7446"}
Oct 30 21:24:47.084628 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.084602    4810 generic.go:356] "Generic (PLEG): container finished" podID="5530ac61-52fa-400c-9506-5a7ea1edb7db" containerID="8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604" exitCode=0
Oct 30 21:24:47.084758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.084659    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"
Oct 30 21:24:47.084758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.084693    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn" event={"ID":"5530ac61-52fa-400c-9506-5a7ea1edb7db","Type":"ContainerDied","Data":"8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604"}
Oct 30 21:24:47.084758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.084736    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn" event={"ID":"5530ac61-52fa-400c-9506-5a7ea1edb7db","Type":"ContainerDied","Data":"7c920e7446d4e1a9e0dbb795f6792e785749d0caa9de2d07ad728f52175b4aa2"}
Oct 30 21:24:47.084758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.084751    4810 scope.go:122] "RemoveContainer" containerID="8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604"
Oct 30 21:24:47.097263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.097242    4810 scope.go:122] "RemoveContainer" containerID="8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604"
Oct 30 21:24:47.097554 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:24:47.097534    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604\": container with ID starting with 8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604 not found: ID does not exist" containerID="8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604"
Oct 30 21:24:47.097634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.097566    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604"} err="failed to get container status \"8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604\": rpc error: code = NotFound desc = could not find container \"8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604\": container with ID starting with 8003c175078832a135c2deb2f94ee01a7c207c0334ae56ca2d2eebd24c76a604 not found: ID does not exist"
Oct 30 21:24:47.122600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.122574    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"]
Oct 30 21:24:47.127685 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.127662    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a574c2-b7c7776ff-7zjsn"]
Oct 30 21:24:47.664404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:47.664372    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5530ac61-52fa-400c-9506-5a7ea1edb7db" path="/var/lib/kubelet/pods/5530ac61-52fa-400c-9506-5a7ea1edb7db/volumes"
Oct 30 21:24:48.300261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:48.300237    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:48.479655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:48.479618    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-xks-ca\") pod \"29b337e7-8252-4971-a190-417b0e4d9bca\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") "
Oct 30 21:24:48.479933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:48.479840    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-openshift-ca\") pod \"29b337e7-8252-4971-a190-417b0e4d9bca\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") "
Oct 30 21:24:48.480114 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:48.480091    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b337e7-8252-4971-a190-417b0e4d9bca-kube-api-access-rggm9\" (UniqueName: \"kubernetes.io/projected/29b337e7-8252-4971-a190-417b0e4d9bca-kube-api-access-rggm9\") pod \"29b337e7-8252-4971-a190-417b0e4d9bca\" (UID: \"29b337e7-8252-4971-a190-417b0e4d9bca\") "
Oct 30 21:24:48.480559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:48.480359    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-openshift-ca" pod "29b337e7-8252-4971-a190-417b0e4d9bca" (UID: "29b337e7-8252-4971-a190-417b0e4d9bca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:48.480788 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:48.480737    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:48.484214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:48.482960    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b337e7-8252-4971-a190-417b0e4d9bca-kube-api-access-rggm9" pod "29b337e7-8252-4971-a190-417b0e4d9bca" (UID: "29b337e7-8252-4971-a190-417b0e4d9bca"). InnerVolumeSpecName "kube-api-access-rggm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:48.484214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:48.483124    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-xks-ca" pod "29b337e7-8252-4971-a190-417b0e4d9bca" (UID: "29b337e7-8252-4971-a190-417b0e4d9bca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:48.581702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:48.581674    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rggm9\" (UniqueName: \"kubernetes.io/projected/29b337e7-8252-4971-a190-417b0e4d9bca-kube-api-access-rggm9\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:48.581702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:48.581701    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b337e7-8252-4971-a190-417b0e4d9bca-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:49.105413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:49.105385    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"
Oct 30 21:24:49.105592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:49.105422    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8" event={"ID":"29b337e7-8252-4971-a190-417b0e4d9bca","Type":"ContainerDied","Data":"31ecf6a82d649539e028e7bb81fa0dd3bc2e212002f76bdb70375039a47cde67"}
Oct 30 21:24:49.105592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:49.105469    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31ecf6a82d649539e028e7bb81fa0dd3bc2e212002f76bdb70375039a47cde67"
Oct 30 21:24:50.784129 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.784089    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"]
Oct 30 21:24:50.785363 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.785335    4810 cpu_manager.go:414] "containerMap: removing container" podUID="29b337e7-8252-4971-a190-417b0e4d9bca" containerName="revoke-keys"
Oct 30 21:24:50.785363 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.785361    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b337e7-8252-4971-a190-417b0e4d9bca" containerName="revoke-keys"
Oct 30 21:24:50.785563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.785452    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5530ac61-52fa-400c-9506-5a7ea1edb7db" containerName="payload-pre-processing"
Oct 30 21:24:50.785563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.785461    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5530ac61-52fa-400c-9506-5a7ea1edb7db" containerName="payload-pre-processing"
Oct 30 21:24:50.785679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.785669    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b337e7-8252-4971-a190-417b0e4d9bca" containerName="revoke-keys"
Oct 30 21:24:50.785739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.785694    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5530ac61-52fa-400c-9506-5a7ea1edb7db" containerName="payload-pre-processing"
Oct 30 21:24:50.791623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.791601    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:24:50.803917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.803845    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a3fb70\""
Oct 30 21:24:50.804042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.803942    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a3fb70-dockercfg-r2k7j\""
Oct 30 21:24:50.806242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.806218    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"]
Oct 30 21:24:50.843994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.843959    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"]
Oct 30 21:24:50.848714 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.848687    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:24:50.854412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.854387    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"]
Oct 30 21:24:50.936491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.935028    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d968\" (UniqueName: \"kubernetes.io/projected/275a0d03-1705-464c-aff5-4b5b8fb668fe-kube-api-access-2d968\") pod \"payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs\" (UID: \"275a0d03-1705-464c-aff5-4b5b8fb668fe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:24:50.936491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:50.935113    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275a0d03-1705-464c-aff5-4b5b8fb668fe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs\" (UID: \"275a0d03-1705-464c-aff5-4b5b8fb668fe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:24:51.040948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.038219    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/029c21db-1fbc-4a68-a513-1fc9da53d370-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz\" (UID: \"029c21db-1fbc-4a68-a513-1fc9da53d370\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:24:51.040948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.038474    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d968\" (UniqueName: \"kubernetes.io/projected/275a0d03-1705-464c-aff5-4b5b8fb668fe-kube-api-access-2d968\") pod \"payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs\" (UID: \"275a0d03-1705-464c-aff5-4b5b8fb668fe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:24:51.040948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.038531    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275a0d03-1705-464c-aff5-4b5b8fb668fe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs\" (UID: \"275a0d03-1705-464c-aff5-4b5b8fb668fe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:24:51.040948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.038603    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv2vk\" (UniqueName: \"kubernetes.io/projected/029c21db-1fbc-4a68-a513-1fc9da53d370-kube-api-access-tv2vk\") pod \"payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz\" (UID: \"029c21db-1fbc-4a68-a513-1fc9da53d370\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:24:51.054213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.053985    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275a0d03-1705-464c-aff5-4b5b8fb668fe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs\" (UID: \"275a0d03-1705-464c-aff5-4b5b8fb668fe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:24:51.070548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.070435    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d968\" (UniqueName: \"kubernetes.io/projected/275a0d03-1705-464c-aff5-4b5b8fb668fe-kube-api-access-2d968\") pod \"payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs\" (UID: \"275a0d03-1705-464c-aff5-4b5b8fb668fe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:24:51.124136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.124102    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:24:51.140071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.139713    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv2vk\" (UniqueName: \"kubernetes.io/projected/029c21db-1fbc-4a68-a513-1fc9da53d370-kube-api-access-tv2vk\") pod \"payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz\" (UID: \"029c21db-1fbc-4a68-a513-1fc9da53d370\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:24:51.140071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.139888    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/029c21db-1fbc-4a68-a513-1fc9da53d370-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz\" (UID: \"029c21db-1fbc-4a68-a513-1fc9da53d370\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:24:51.140760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.140732    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/029c21db-1fbc-4a68-a513-1fc9da53d370-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz\" (UID: \"029c21db-1fbc-4a68-a513-1fc9da53d370\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:24:51.150067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.150041    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv2vk\" (UniqueName: \"kubernetes.io/projected/029c21db-1fbc-4a68-a513-1fc9da53d370-kube-api-access-tv2vk\") pod \"payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz\" (UID: \"029c21db-1fbc-4a68-a513-1fc9da53d370\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:24:51.215815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.215761    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:24:51.425885 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:24:51.425843    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275a0d03_1705_464c_aff5_4b5b8fb668fe.slice/crio-f9883eed6704509294988eb2b530d3806f773a423b205431ba5a0e08d8c2cd9b WatchSource:0}: Error finding container f9883eed6704509294988eb2b530d3806f773a423b205431ba5a0e08d8c2cd9b: Status 404 returned error can't find the container with id f9883eed6704509294988eb2b530d3806f773a423b205431ba5a0e08d8c2cd9b
Oct 30 21:24:51.426000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.425967    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"]
Oct 30 21:24:51.534459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:51.534432    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"]
Oct 30 21:24:51.535307 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:24:51.535281    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029c21db_1fbc_4a68_a513_1fc9da53d370.slice/crio-cb82f853598a45971d63b1bfe16ac4d475b47c13de173e3943244168d66a9179 WatchSource:0}: Error finding container cb82f853598a45971d63b1bfe16ac4d475b47c13de173e3943244168d66a9179: Status 404 returned error can't find the container with id cb82f853598a45971d63b1bfe16ac4d475b47c13de173e3943244168d66a9179
Oct 30 21:24:52.133364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:52.133326    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs" event={"ID":"275a0d03-1705-464c-aff5-4b5b8fb668fe","Type":"ContainerStarted","Data":"850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953"}
Oct 30 21:24:52.133983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:52.133383    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs" event={"ID":"275a0d03-1705-464c-aff5-4b5b8fb668fe","Type":"ContainerStarted","Data":"f9883eed6704509294988eb2b530d3806f773a423b205431ba5a0e08d8c2cd9b"}
Oct 30 21:24:52.133983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:52.133413    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:24:52.135144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:52.135116    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz" event={"ID":"029c21db-1fbc-4a68-a513-1fc9da53d370","Type":"ContainerStarted","Data":"4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783"}
Oct 30 21:24:52.135250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:52.135168    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz" event={"ID":"029c21db-1fbc-4a68-a513-1fc9da53d370","Type":"ContainerStarted","Data":"cb82f853598a45971d63b1bfe16ac4d475b47c13de173e3943244168d66a9179"}
Oct 30 21:24:52.135250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:52.135200    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:24:52.150102 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:52.150063    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs" podStartSLOduration=2.150050926 podStartE2EDuration="2.150050926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:52.148075189 +0000 UTC m=+2858.639891745" watchObservedRunningTime="2026-10-30 21:24:52.150050926 +0000 UTC m=+2858.641867469"
Oct 30 21:24:52.166186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:52.166142    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz" podStartSLOduration=2.166130372 podStartE2EDuration="2.166130372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:52.162813979 +0000 UTC m=+2858.654630551" watchObservedRunningTime="2026-10-30 21:24:52.166130372 +0000 UTC m=+2858.657946915"
Oct 30 21:24:53.281516 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:24:53.281484    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:24:55.160742 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:24:55.160697    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:25:03.174263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:03.174228    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:25:03.174705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:03.174318    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:25:04.418569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:04.418480    4810 generic.go:356] "Generic (PLEG): container finished" podID="9d205dba-ff07-4742-8509-44948ba1af95" containerID="07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73" exitCode=0
Oct 30 21:25:04.419121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:04.418669    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" event={"ID":"9d205dba-ff07-4742-8509-44948ba1af95","Type":"ContainerDied","Data":"07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73"}
Oct 30 21:25:04.419262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:04.419242    4810 scope.go:122] "RemoveContainer" containerID="07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73"
Oct 30 21:25:04.425463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:04.425168    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:04.425463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:04.425258    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:04.425463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:04.425301    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:05.430518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.430472    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" event={"ID":"9d205dba-ff07-4742-8509-44948ba1af95","Type":"ContainerStarted","Data":"98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e"}
Oct 30 21:25:05.720946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.720846    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"]
Oct 30 21:25:05.727679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.726270    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.734339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.733587    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ddcf93\"/\"openshift-service-ca.crt\""
Oct 30 21:25:05.734339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.733705    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ddcf93\"/\"test-model-ddcf93-kserve-self-signed-certs\""
Oct 30 21:25:05.734339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.733767    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ddcf93\"/\"kube-root-ca.crt\""
Oct 30 21:25:05.734339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.733872    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ddcf93\"/\"default-dockercfg-trm4s\""
Oct 30 21:25:05.747025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.738809    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"]
Oct 30 21:25:05.770483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.770437    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8npfr\" (UniqueName: \"kubernetes.io/projected/d48ab868-05de-46c7-89ba-14468f8d778b-kube-api-access-8npfr\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.770664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.770613    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-tmp-dir\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.770761 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.770741    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-model-cache\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.775884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.770957    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-dshm\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.775884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.771021    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48ab868-05de-46c7-89ba-14468f8d778b-tls-certs\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.775884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.771187    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-home\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.895880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.895459    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-tmp-dir\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.895880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.895530    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-model-cache\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.895880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.895683    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-dshm\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.895880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.895733    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48ab868-05de-46c7-89ba-14468f8d778b-tls-certs\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.895880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.895819    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-home\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.896345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.895966    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8npfr\" (UniqueName: \"kubernetes.io/projected/d48ab868-05de-46c7-89ba-14468f8d778b-kube-api-access-8npfr\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.906877 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.896807    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-tmp-dir\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.906877 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.897104    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-model-cache\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.907497 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.907466    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-home\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.923879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.921920    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8npfr\" (UniqueName: \"kubernetes.io/projected/d48ab868-05de-46c7-89ba-14468f8d778b-kube-api-access-8npfr\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.941881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.929823    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-dshm\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:05.976930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:05.970906    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48ab868-05de-46c7-89ba-14468f8d778b-tls-certs\") pod \"test-model-ddcf93-kserve-b98c95d45-n4ng6\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") " pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:06.070171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:06.070135    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:06.302881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:06.282897    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:25:06.365949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:06.326706    4810 patch_prober.go:29] interesting pod/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Oct 30 21:25:06.365949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:06.326763    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Oct 30 21:25:06.662000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:06.661969    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"]
Oct 30 21:25:07.283884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:07.277191    4810 patch_prober.go:29] interesting pod/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Oct 30 21:25:07.283884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:07.277249    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Oct 30 21:25:07.494881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:07.493136    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6" event={"ID":"d48ab868-05de-46c7-89ba-14468f8d778b","Type":"ContainerStarted","Data":"0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3"}
Oct 30 21:25:07.494881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:07.493208    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6" event={"ID":"d48ab868-05de-46c7-89ba-14468f8d778b","Type":"ContainerStarted","Data":"1f1bbe2690423c459556dfa27db7fe7454a8b5bd9ea9b8ac39766bd16108bab7"}
Oct 30 21:25:07.734163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:07.734131    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:25:08.070816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:08.070747    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:08.287890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:08.287646    4810 patch_prober.go:29] interesting pod/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Oct 30 21:25:08.287890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:08.287704    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Oct 30 21:25:09.277138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:09.277100    4810 patch_prober.go:29] interesting pod/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Oct 30 21:25:09.277777 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:09.277743    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Oct 30 21:25:10.284882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:10.276185    4810 patch_prober.go:29] interesting pod/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Oct 30 21:25:10.284882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:10.276258    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Oct 30 21:25:11.281738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:11.281683    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:25:11.283179 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:11.283153    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:25:11.305968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:11.305921    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6" podStartSLOduration=6.305908688 podStartE2EDuration="6.305908688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:07.517054898 +0000 UTC m=+2874.008871447" watchObservedRunningTime="2026-10-30 21:25:11.305908688 +0000 UTC m=+2877.797725227"
Oct 30 21:25:15.168898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.168846    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"]
Oct 30 21:25:15.173378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.173350    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-176db5-j7tv8"]
Oct 30 21:25:15.668507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.668471    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b337e7-8252-4971-a190-417b0e4d9bca" path="/var/lib/kubelet/pods/29b337e7-8252-4971-a190-417b0e4d9bca/volumes"
Oct 30 21:25:15.895936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.895900    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"]
Oct 30 21:25:15.900065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.900044    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:15.930429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.930339    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5506f4\"/\"default-dockercfg-vsrd6\""
Oct 30 21:25:15.930605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.930470    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5506f4\"/\"kube-root-ca.crt\""
Oct 30 21:25:15.931064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.931023    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5506f4\"/\"openshift-service-ca.crt\""
Oct 30 21:25:15.931189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.931168    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5506f4\"/\"test-model-5506f4-kserve-self-signed-certs\""
Oct 30 21:25:15.935896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.935843    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"]
Oct 30 21:25:15.997217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.997182    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tmp-dir\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:15.997217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.997227    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tls-certs\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:15.997473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.997452    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzt6p\" (UniqueName: \"kubernetes.io/projected/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-kube-api-access-zzt6p\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:15.997535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.997505    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-model-cache\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:15.997712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.997693    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-home\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:15.997775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:15.997735    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-dshm\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.070923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.070872    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:16.089950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.089919    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:16.098567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.098542    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzt6p\" (UniqueName: \"kubernetes.io/projected/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-kube-api-access-zzt6p\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.098743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.098603    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-model-cache\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.098743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.098728    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-home\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.098880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.098766    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-dshm\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.098963 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.098943    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tmp-dir\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.099029 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.098975    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tls-certs\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.099686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.099181    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-model-cache\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.099686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.099639    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tmp-dir\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.099915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.099852    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-home\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.101722 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.101700    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-dshm\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.101897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.101831    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tls-certs\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.171133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.153167    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzt6p\" (UniqueName: \"kubernetes.io/projected/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-kube-api-access-zzt6p\") pod \"test-model-5506f4-kserve-658584bdc9-vwqxt\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") " pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.249905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.249160    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:16.602435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.602349    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:25:16.638793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:16.638754    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"]
Oct 30 21:25:17.602277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:17.602238    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt" event={"ID":"a7e8d92d-252b-4abb-ab86-3efc8c0759a1","Type":"ContainerStarted","Data":"93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c"}
Oct 30 21:25:17.602770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:17.602291    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt" event={"ID":"a7e8d92d-252b-4abb-ab86-3efc8c0759a1","Type":"ContainerStarted","Data":"6e4b848231d8f6e61b30a565b966b1fd36deb5fa8e97e15a4721b649107d887b"}
Oct 30 21:25:17.624963 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:17.624898    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt" podStartSLOduration=2.624882556 podStartE2EDuration="2.624882556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:17.620287113 +0000 UTC m=+2884.112103658" watchObservedRunningTime="2026-10-30 21:25:17.624882556 +0000 UTC m=+2884.116699099"
Oct 30 21:25:18.249965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:18.249932    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:18.875699 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:18.875671    4810 scope.go:122] "RemoveContainer" containerID="fba7fefd5295d7ec8beffdf11b141585864369106764232e23624a0a898c683c"
Oct 30 21:25:20.071389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.071343    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"]
Oct 30 21:25:20.084336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.083773    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.089061 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.088991    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"]
Oct 30 21:25:20.090282 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.090176    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ffce13\"/\"openshift-service-ca.crt\""
Oct 30 21:25:20.090282 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.090253    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ffce13\"/\"kube-root-ca.crt\""
Oct 30 21:25:20.091084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.091061    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ffce13\"/\"default-dockercfg-dkxml\""
Oct 30 21:25:20.091203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.091140    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ffce13\"/\"test-model-ffce13-kserve-self-signed-certs\""
Oct 30 21:25:20.183678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.183635    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-home\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.183965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.183706    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-dshm\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.183965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.183907    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-model-cache\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.183965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.183947    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb4beb5-9cef-4b11-add6-c18470c22c09-tls-certs\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.184167 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.184016    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-tmp-dir\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.184167 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.184064    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xk86\" (UniqueName: \"kubernetes.io/projected/3fb4beb5-9cef-4b11-add6-c18470c22c09-kube-api-access-7xk86\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.293924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.289661    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-home\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.293924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.289723    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-dshm\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.293924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.289904    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-model-cache\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.293924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.289938    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb4beb5-9cef-4b11-add6-c18470c22c09-tls-certs\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.293924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.290009    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-tmp-dir\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.293924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.290161    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xk86\" (UniqueName: \"kubernetes.io/projected/3fb4beb5-9cef-4b11-add6-c18470c22c09-kube-api-access-7xk86\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.293924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.291207    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-home\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.305071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.305042    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-tmp-dir\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.305277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.305013    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-model-cache\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.399559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.399523    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb4beb5-9cef-4b11-add6-c18470c22c09-tls-certs\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.413878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.413249    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-dshm\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.441891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.431199    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xk86\" (UniqueName: \"kubernetes.io/projected/3fb4beb5-9cef-4b11-add6-c18470c22c09-kube-api-access-7xk86\") pod \"test-model-ffce13-kserve-5d595c6db7-j624t\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") " pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:20.719538 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:20.719463    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:21.010905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:21.010808    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"]
Oct 30 21:25:21.016456 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:25:21.016419    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb4beb5_9cef_4b11_add6_c18470c22c09.slice/crio-4338eba1af8bc59ecf97cffa197d45a3eb979cef7f7929d4adbdeb810f62f0a5 WatchSource:0}: Error finding container 4338eba1af8bc59ecf97cffa197d45a3eb979cef7f7929d4adbdeb810f62f0a5: Status 404 returned error can't find the container with id 4338eba1af8bc59ecf97cffa197d45a3eb979cef7f7929d4adbdeb810f62f0a5
Oct 30 21:25:21.642226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:21.642188    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t" event={"ID":"3fb4beb5-9cef-4b11-add6-c18470c22c09","Type":"ContainerStarted","Data":"67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c"}
Oct 30 21:25:21.642635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:21.642236    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t" event={"ID":"3fb4beb5-9cef-4b11-add6-c18470c22c09","Type":"ContainerStarted","Data":"4338eba1af8bc59ecf97cffa197d45a3eb979cef7f7929d4adbdeb810f62f0a5"}
Oct 30 21:25:21.673644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:21.673554    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t" podStartSLOduration=1.673535406 podStartE2EDuration="1.673535406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:21.671322541 +0000 UTC m=+2888.163139084" watchObservedRunningTime="2026-10-30 21:25:21.673535406 +0000 UTC m=+2888.165351949"
Oct 30 21:25:21.719598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:21.719562    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:23.869065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:23.869032    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"]
Oct 30 21:25:23.874964 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:23.874942    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:23.885389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:23.885302    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:25:23.885389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:23.885363    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:23.902759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:23.902727    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"]
Oct 30 21:25:23.963383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:23.963355    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:23.963537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:23.963390    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-727vl\" (UniqueName: \"kubernetes.io/projected/e3aebdb9-7c0a-487b-9630-11eecedf33db-kube-api-access-727vl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:23.963586 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:23.963533    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:24.064938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:24.064906    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:24.065102 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:24.065079    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:24.065150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:24.065103    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-727vl\" (UniqueName: \"kubernetes.io/projected/e3aebdb9-7c0a-487b-9630-11eecedf33db-kube-api-access-727vl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:24.065591 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:24.065570    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:24.067528 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:24.067507    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:24.073292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:24.073265    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-727vl\" (UniqueName: \"kubernetes.io/projected/e3aebdb9-7c0a-487b-9630-11eecedf33db-kube-api-access-727vl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:24.197326 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:24.197286    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:24.491271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:24.491223    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"]
Oct 30 21:25:24.491505 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:25:24.491478    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3aebdb9_7c0a_487b_9630_11eecedf33db.slice/crio-0948d60bde2a4419c009b279219d0f018d8d9cbc8fcb75332c86e496292e6198 WatchSource:0}: Error finding container 0948d60bde2a4419c009b279219d0f018d8d9cbc8fcb75332c86e496292e6198: Status 404 returned error can't find the container with id 0948d60bde2a4419c009b279219d0f018d8d9cbc8fcb75332c86e496292e6198
Oct 30 21:25:24.671366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:24.671338    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn" event={"ID":"e3aebdb9-7c0a-487b-9630-11eecedf33db","Type":"ContainerStarted","Data":"0948d60bde2a4419c009b279219d0f018d8d9cbc8fcb75332c86e496292e6198"}
Oct 30 21:25:25.157901 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:25:25.157839    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:25:25.684762 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:25.684651    4810 generic.go:356] "Generic (PLEG): container finished" podID="e3aebdb9-7c0a-487b-9630-11eecedf33db" containerID="631e9a87184784f9231f9f598bef46ffee5a6f29f22478ed9d90081117c9c611" exitCode=0
Oct 30 21:25:25.684762 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:25.684746    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn" event={"ID":"e3aebdb9-7c0a-487b-9630-11eecedf33db","Type":"ContainerDied","Data":"631e9a87184784f9231f9f598bef46ffee5a6f29f22478ed9d90081117c9c611"}
Oct 30 21:25:26.220346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.211651    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"]
Oct 30 21:25:26.225325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.225295    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"]
Oct 30 21:25:26.226249 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.226224    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.232696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.232667    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9b78e4\"/\"openshift-service-ca.crt\""
Oct 30 21:25:26.232847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.232831    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9b78e4\"/\"default-dockercfg-9q82c\""
Oct 30 21:25:26.233010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.232972    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9b78e4\"/\"kube-root-ca.crt\""
Oct 30 21:25:26.233173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.233156    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9b78e4\"/\"test-model-9b78e4-kserve-self-signed-certs\""
Oct 30 21:25:26.250984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.250961    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:26.269612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.269592    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:26.307881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.307777    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-model-cache\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.308062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.307906    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-home\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.308062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.307951    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-tmp-dir\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.308195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.308095    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130112da-8af4-44a1-99bf-4728f09414d0-tls-certs\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.308255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.308192    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnx5m\" (UniqueName: \"kubernetes.io/projected/130112da-8af4-44a1-99bf-4728f09414d0-kube-api-access-xnx5m\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.308357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.308336    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-dshm\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.415878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.414845    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130112da-8af4-44a1-99bf-4728f09414d0-tls-certs\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.415878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.414973    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnx5m\" (UniqueName: \"kubernetes.io/projected/130112da-8af4-44a1-99bf-4728f09414d0-kube-api-access-xnx5m\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.415878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.415085    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-dshm\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.415878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.415267    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-model-cache\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.415878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.415312    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-home\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.415878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.415351    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-tmp-dir\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.432880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.418205    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-home\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.432880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.418653    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-model-cache\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.432880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.418927    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-tmp-dir\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.432880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.426450    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130112da-8af4-44a1-99bf-4728f09414d0-tls-certs\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.432880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.427238    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-dshm\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.442886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.436472    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnx5m\" (UniqueName: \"kubernetes.io/projected/130112da-8af4-44a1-99bf-4728f09414d0-kube-api-access-xnx5m\") pod \"test-model-9b78e4-kserve-99676f64f-xx7lq\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") " pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.550878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.550780    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:26.719069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.719037    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:25:26.880478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.880438    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"]
Oct 30 21:25:26.881554 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:25:26.881506    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod130112da_8af4_44a1_99bf_4728f09414d0.slice/crio-1dd8decbfe5e14327240c973675ec44a9f317ddf632f8af5ea3eae82dafcdfae WatchSource:0}: Error finding container 1dd8decbfe5e14327240c973675ec44a9f317ddf632f8af5ea3eae82dafcdfae: Status 404 returned error can't find the container with id 1dd8decbfe5e14327240c973675ec44a9f317ddf632f8af5ea3eae82dafcdfae
Oct 30 21:25:26.909954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:26.909932    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:27.029708 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.029678    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3aebdb9-7c0a-487b-9630-11eecedf33db-kube-api-access-727vl\" (UniqueName: \"kubernetes.io/projected/e3aebdb9-7c0a-487b-9630-11eecedf33db-kube-api-access-727vl\") pod \"e3aebdb9-7c0a-487b-9630-11eecedf33db\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") "
Oct 30 21:25:27.029897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.029737    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-openshift-ca\") pod \"e3aebdb9-7c0a-487b-9630-11eecedf33db\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") "
Oct 30 21:25:27.029897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.029890    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-xks-ca\") pod \"e3aebdb9-7c0a-487b-9630-11eecedf33db\" (UID: \"e3aebdb9-7c0a-487b-9630-11eecedf33db\") "
Oct 30 21:25:27.031153 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.030587    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-openshift-ca" pod "e3aebdb9-7c0a-487b-9630-11eecedf33db" (UID: "e3aebdb9-7c0a-487b-9630-11eecedf33db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:27.032314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.032286    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aebdb9-7c0a-487b-9630-11eecedf33db-kube-api-access-727vl" pod "e3aebdb9-7c0a-487b-9630-11eecedf33db" (UID: "e3aebdb9-7c0a-487b-9630-11eecedf33db"). InnerVolumeSpecName "kube-api-access-727vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:27.032512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.032489    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-xks-ca" pod "e3aebdb9-7c0a-487b-9630-11eecedf33db" (UID: "e3aebdb9-7c0a-487b-9630-11eecedf33db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:27.137958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.133411    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-727vl\" (UniqueName: \"kubernetes.io/projected/e3aebdb9-7c0a-487b-9630-11eecedf33db-kube-api-access-727vl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:27.137958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.133447    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:27.137958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.133462    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3aebdb9-7c0a-487b-9630-11eecedf33db-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:27.712505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.712415    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq" event={"ID":"130112da-8af4-44a1-99bf-4728f09414d0","Type":"ContainerStarted","Data":"664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270"}
Oct 30 21:25:27.712505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.712463    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq" event={"ID":"130112da-8af4-44a1-99bf-4728f09414d0","Type":"ContainerStarted","Data":"1dd8decbfe5e14327240c973675ec44a9f317ddf632f8af5ea3eae82dafcdfae"}
Oct 30 21:25:27.714290 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.714258    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"
Oct 30 21:25:27.714438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.714284    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn" event={"ID":"e3aebdb9-7c0a-487b-9630-11eecedf33db","Type":"ContainerDied","Data":"0948d60bde2a4419c009b279219d0f018d8d9cbc8fcb75332c86e496292e6198"}
Oct 30 21:25:27.714438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.714326    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0948d60bde2a4419c009b279219d0f018d8d9cbc8fcb75332c86e496292e6198"
Oct 30 21:25:27.737729 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:27.737678    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq" podStartSLOduration=1.737661733 podStartE2EDuration="1.737661733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:27.732083246 +0000 UTC m=+2894.223899794" watchObservedRunningTime="2026-10-30 21:25:27.737661733 +0000 UTC m=+2894.229478279"
Oct 30 21:25:28.551377 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:28.551340    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:30.720405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:30.720365    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:30.738251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:30.738223    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:30.821029 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:30.820996    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:25:31.684517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:31.684478    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:25:33.993391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:33.993358    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"]
Oct 30 21:25:34.018910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:34.017059    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"]
Oct 30 21:25:36.552124 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:36.552078    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:36.566439 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:36.566413    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:36.890591 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:36.890554    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:25:37.338584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.338510    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"]
Oct 30 21:25:37.339386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.339369    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e3aebdb9-7c0a-487b-9630-11eecedf33db" containerName="revoke-keys"
Oct 30 21:25:37.339435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.339388    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3aebdb9-7c0a-487b-9630-11eecedf33db" containerName="revoke-keys"
Oct 30 21:25:37.339541 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.339532    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3aebdb9-7c0a-487b-9630-11eecedf33db" containerName="revoke-keys"
Oct 30 21:25:37.344958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.344937    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:37.349324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.349304    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:37.349689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.349670    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:25:37.355000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.354350    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"]
Oct 30 21:25:37.443206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.443167    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5ff4a45-qt5rp\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:37.443387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.443227    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5w8g\" (UniqueName: \"kubernetes.io/projected/82da14a4-2ad0-4915-a4ac-4e46a32d5586-kube-api-access-r5w8g\") pod \"maas-api-revoke-sub-c5ff4a45-qt5rp\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:37.443487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.443467    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5ff4a45-qt5rp\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:37.544606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.544562    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5ff4a45-qt5rp\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:37.544805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.544623    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5w8g\" (UniqueName: \"kubernetes.io/projected/82da14a4-2ad0-4915-a4ac-4e46a32d5586-kube-api-access-r5w8g\") pod \"maas-api-revoke-sub-c5ff4a45-qt5rp\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:37.544805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.544763    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5ff4a45-qt5rp\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:37.545412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.545389    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5ff4a45-qt5rp\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:37.547006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.546989    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5ff4a45-qt5rp\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:37.552769 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.552751    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5w8g\" (UniqueName: \"kubernetes.io/projected/82da14a4-2ad0-4915-a4ac-4e46a32d5586-kube-api-access-r5w8g\") pod \"maas-api-revoke-sub-c5ff4a45-qt5rp\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:37.664422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:37.664394    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:38.004007 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:25:38.003976    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82da14a4_2ad0_4915_a4ac_4e46a32d5586.slice/crio-4ed8daa1cb8701dc71445a52ae6e58f315c5dadc6de868d551ab9e53779df15c WatchSource:0}: Error finding container 4ed8daa1cb8701dc71445a52ae6e58f315c5dadc6de868d551ab9e53779df15c: Status 404 returned error can't find the container with id 4ed8daa1cb8701dc71445a52ae6e58f315c5dadc6de868d551ab9e53779df15c
Oct 30 21:25:38.005749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:38.005721    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"]
Oct 30 21:25:38.898492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:38.898453    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp" event={"ID":"82da14a4-2ad0-4915-a4ac-4e46a32d5586","Type":"ContainerStarted","Data":"9998550cf3be3ef6344bdcaf6c6d586c1c9edc2fc860351a3b107d30af2bb10d"}
Oct 30 21:25:38.898959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:38.898516    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp" event={"ID":"82da14a4-2ad0-4915-a4ac-4e46a32d5586","Type":"ContainerStarted","Data":"4ed8daa1cb8701dc71445a52ae6e58f315c5dadc6de868d551ab9e53779df15c"}
Oct 30 21:25:38.917974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:38.917924    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp" podStartSLOduration=1.9179070280000001 podStartE2EDuration="1.917907028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:38.915719158 +0000 UTC m=+2905.407535702" watchObservedRunningTime="2026-10-30 21:25:38.917907028 +0000 UTC m=+2905.409723572"
Oct 30 21:25:38.994545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:38.994510    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm" podUID="7e70ac49-535d-4ba4-af02-4f15b3e4ed5b" containerName="payload-processing" containerID="cri-o://5c190ee7d3a948ae65fd41b6fbcb9fd0796ba304739be52a757dbe5c6b55b2c8" gracePeriod=25
Oct 30 21:25:39.017516 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:39.017475    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l" podUID="e4f5604e-38c2-4f39-9f51-496b1a383cc9" containerName="payload-pre-processing" containerID="cri-o://b660eaaf20d9eff2dd70c643e0409caee011b9da0275a2a2dac2836b62178cc1" gracePeriod=25
Oct 30 21:25:39.911820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:39.911763    4810 generic.go:356] "Generic (PLEG): container finished" podID="82da14a4-2ad0-4915-a4ac-4e46a32d5586" containerID="9998550cf3be3ef6344bdcaf6c6d586c1c9edc2fc860351a3b107d30af2bb10d" exitCode=0
Oct 30 21:25:39.912266 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:39.911850    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp" event={"ID":"82da14a4-2ad0-4915-a4ac-4e46a32d5586","Type":"ContainerDied","Data":"9998550cf3be3ef6344bdcaf6c6d586c1c9edc2fc860351a3b107d30af2bb10d"}
Oct 30 21:25:41.233742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.233713    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:41.322891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.322848    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-xks-ca\") pod \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") "
Oct 30 21:25:41.323087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.322995    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-openshift-ca\") pod \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") "
Oct 30 21:25:41.323160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.323107    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82da14a4-2ad0-4915-a4ac-4e46a32d5586-kube-api-access-r5w8g\" (UniqueName: \"kubernetes.io/projected/82da14a4-2ad0-4915-a4ac-4e46a32d5586-kube-api-access-r5w8g\") pod \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\" (UID: \"82da14a4-2ad0-4915-a4ac-4e46a32d5586\") "
Oct 30 21:25:41.323751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.323694    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-openshift-ca" pod "82da14a4-2ad0-4915-a4ac-4e46a32d5586" (UID: "82da14a4-2ad0-4915-a4ac-4e46a32d5586"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:41.325227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.325207    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-xks-ca" pod "82da14a4-2ad0-4915-a4ac-4e46a32d5586" (UID: "82da14a4-2ad0-4915-a4ac-4e46a32d5586"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:41.326575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.326554    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82da14a4-2ad0-4915-a4ac-4e46a32d5586-kube-api-access-r5w8g" pod "82da14a4-2ad0-4915-a4ac-4e46a32d5586" (UID: "82da14a4-2ad0-4915-a4ac-4e46a32d5586"). InnerVolumeSpecName "kube-api-access-r5w8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:41.425745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.425708    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5w8g\" (UniqueName: \"kubernetes.io/projected/82da14a4-2ad0-4915-a4ac-4e46a32d5586-kube-api-access-r5w8g\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.425745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.425745    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.425981 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.425759    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82da14a4-2ad0-4915-a4ac-4e46a32d5586-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.445634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.445565    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm" podUID="7e70ac49-535d-4ba4-af02-4f15b3e4ed5b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:25:41.445758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.445565    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l" podUID="e4f5604e-38c2-4f39-9f51-496b1a383cc9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:25:41.941606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.941576    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp" event={"ID":"82da14a4-2ad0-4915-a4ac-4e46a32d5586","Type":"ContainerDied","Data":"4ed8daa1cb8701dc71445a52ae6e58f315c5dadc6de868d551ab9e53779df15c"}
Oct 30 21:25:41.941820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.941810    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ed8daa1cb8701dc71445a52ae6e58f315c5dadc6de868d551ab9e53779df15c"
Oct 30 21:25:41.942131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:41.941990    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"
Oct 30 21:25:46.573471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:46.573436    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"]
Oct 30 21:25:46.578147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:46.578113    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-74f9f4-dxlgn"]
Oct 30 21:25:47.664500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:47.664468    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3aebdb9-7c0a-487b-9630-11eecedf33db" path="/var/lib/kubelet/pods/e3aebdb9-7c0a-487b-9630-11eecedf33db/volumes"
Oct 30 21:25:47.961085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:47.961010    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"]
Oct 30 21:25:47.961921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:47.961905    4810 cpu_manager.go:414] "containerMap: removing container" podUID="82da14a4-2ad0-4915-a4ac-4e46a32d5586" containerName="revoke-keys"
Oct 30 21:25:47.961979 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:47.961922    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82da14a4-2ad0-4915-a4ac-4e46a32d5586" containerName="revoke-keys"
Oct 30 21:25:47.962062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:47.962053    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="82da14a4-2ad0-4915-a4ac-4e46a32d5586" containerName="revoke-keys"
Oct 30 21:25:47.970497 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:47.970444    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:47.982476 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:47.981670    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:25:47.982476 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:47.981755    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:47.993772 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:47.993745    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"]
Oct 30 21:25:48.054870 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.054826    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06d3331e-tw97l\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:48.055014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.054899    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2jm\" (UniqueName: \"kubernetes.io/projected/aded187d-2e01-42b7-9f54-8279aa364576-kube-api-access-hd2jm\") pod \"maas-api-revoke-sub-06d3331e-tw97l\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:48.055126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.055099    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06d3331e-tw97l\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:48.158871 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.158812    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06d3331e-tw97l\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:48.159381 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.159354    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06d3331e-tw97l\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:48.159540 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.159525    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2jm\" (UniqueName: \"kubernetes.io/projected/aded187d-2e01-42b7-9f54-8279aa364576-kube-api-access-hd2jm\") pod \"maas-api-revoke-sub-06d3331e-tw97l\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:48.171878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.160796    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06d3331e-tw97l\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:48.217978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.217844    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2jm\" (UniqueName: \"kubernetes.io/projected/aded187d-2e01-42b7-9f54-8279aa364576-kube-api-access-hd2jm\") pod \"maas-api-revoke-sub-06d3331e-tw97l\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:48.226878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.219489    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06d3331e-tw97l\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:48.297036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.296991    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:48.637700 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:25:48.637668    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaded187d_2e01_42b7_9f54_8279aa364576.slice/crio-66a0b13855209ccecc5fa56c6b0e99656b42caac63ec2d1b0666a4c56c0dd710 WatchSource:0}: Error finding container 66a0b13855209ccecc5fa56c6b0e99656b42caac63ec2d1b0666a4c56c0dd710: Status 404 returned error can't find the container with id 66a0b13855209ccecc5fa56c6b0e99656b42caac63ec2d1b0666a4c56c0dd710
Oct 30 21:25:48.642556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:48.642521    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"]
Oct 30 21:25:49.030943 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:49.030895    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l" event={"ID":"aded187d-2e01-42b7-9f54-8279aa364576","Type":"ContainerStarted","Data":"66a0b13855209ccecc5fa56c6b0e99656b42caac63ec2d1b0666a4c56c0dd710"}
Oct 30 21:25:50.043608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:50.043525    4810 generic.go:356] "Generic (PLEG): container finished" podID="aded187d-2e01-42b7-9f54-8279aa364576" containerID="084ae85317acf352a652d4fdf6efe0e796a9f41b42e0bbb054a7d8b0b04d0951" exitCode=0
Oct 30 21:25:50.044073 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:50.043613    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l" event={"ID":"aded187d-2e01-42b7-9f54-8279aa364576","Type":"ContainerDied","Data":"084ae85317acf352a652d4fdf6efe0e796a9f41b42e0bbb054a7d8b0b04d0951"}
Oct 30 21:25:51.294998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.294971    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:51.442565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.442529    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aded187d-2e01-42b7-9f54-8279aa364576-kube-api-access-hd2jm\" (UniqueName: \"kubernetes.io/projected/aded187d-2e01-42b7-9f54-8279aa364576-kube-api-access-hd2jm\") pod \"aded187d-2e01-42b7-9f54-8279aa364576\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") "
Oct 30 21:25:51.442768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.442710    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-openshift-ca\") pod \"aded187d-2e01-42b7-9f54-8279aa364576\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") "
Oct 30 21:25:51.442849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.442815    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-xks-ca\") pod \"aded187d-2e01-42b7-9f54-8279aa364576\" (UID: \"aded187d-2e01-42b7-9f54-8279aa364576\") "
Oct 30 21:25:51.443282 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.443252    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-openshift-ca" pod "aded187d-2e01-42b7-9f54-8279aa364576" (UID: "aded187d-2e01-42b7-9f54-8279aa364576"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:51.443587 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.443546    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:51.446004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.445977    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-xks-ca" pod "aded187d-2e01-42b7-9f54-8279aa364576" (UID: "aded187d-2e01-42b7-9f54-8279aa364576"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:51.446103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.446031    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aded187d-2e01-42b7-9f54-8279aa364576-kube-api-access-hd2jm" pod "aded187d-2e01-42b7-9f54-8279aa364576" (UID: "aded187d-2e01-42b7-9f54-8279aa364576"). InnerVolumeSpecName "kube-api-access-hd2jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:51.446622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.446594    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm" podUID="7e70ac49-535d-4ba4-af02-4f15b3e4ed5b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:25:51.446852 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.446827    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l" podUID="e4f5604e-38c2-4f39-9f51-496b1a383cc9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:25:51.544642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.544606    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd2jm\" (UniqueName: \"kubernetes.io/projected/aded187d-2e01-42b7-9f54-8279aa364576-kube-api-access-hd2jm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:51.544642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:51.544640    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aded187d-2e01-42b7-9f54-8279aa364576-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:52.063444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:52.063351    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l" event={"ID":"aded187d-2e01-42b7-9f54-8279aa364576","Type":"ContainerDied","Data":"66a0b13855209ccecc5fa56c6b0e99656b42caac63ec2d1b0666a4c56c0dd710"}
Oct 30 21:25:52.063444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:52.063412    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a0b13855209ccecc5fa56c6b0e99656b42caac63ec2d1b0666a4c56c0dd710"
Oct 30 21:25:52.063444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:52.063415    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"
Oct 30 21:25:54.889322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.889244    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"]
Oct 30 21:25:54.890163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.890143    4810 cpu_manager.go:414] "containerMap: removing container" podUID="aded187d-2e01-42b7-9f54-8279aa364576" containerName="revoke-keys"
Oct 30 21:25:54.890163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.890163    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aded187d-2e01-42b7-9f54-8279aa364576" containerName="revoke-keys"
Oct 30 21:25:54.890299 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.890290    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="aded187d-2e01-42b7-9f54-8279aa364576" containerName="revoke-keys"
Oct 30 21:25:54.895581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.895495    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:54.900668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.900640    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:25:54.900791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.900688    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:54.903256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.902534    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"]
Oct 30 21:25:54.925344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.924948    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:54.925344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.925091    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:54.925344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:54.925193    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8bxk\" (UniqueName: \"kubernetes.io/projected/70841359-02bc-4f62-a78c-fee5cc05a2f1-kube-api-access-l8bxk\") pod \"maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:55.026291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:55.026252    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:55.026452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:55.026306    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bxk\" (UniqueName: \"kubernetes.io/projected/70841359-02bc-4f62-a78c-fee5cc05a2f1-kube-api-access-l8bxk\") pod \"maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:55.026520 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:55.026505    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:55.027069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:55.027041    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:55.028945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:55.028919    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:55.035207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:55.035181    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8bxk\" (UniqueName: \"kubernetes.io/projected/70841359-02bc-4f62-a78c-fee5cc05a2f1-kube-api-access-l8bxk\") pod \"maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:55.159787 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:25:55.159679    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:25:55.234037 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:55.234001    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:55.525004 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:25:55.524962    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70841359_02bc_4f62_a78c_fee5cc05a2f1.slice/crio-9d8b670089d27a3d0edc29af9f866b6179cb02f1d9f0529a3918de2fa581c585 WatchSource:0}: Error finding container 9d8b670089d27a3d0edc29af9f866b6179cb02f1d9f0529a3918de2fa581c585: Status 404 returned error can't find the container with id 9d8b670089d27a3d0edc29af9f866b6179cb02f1d9f0529a3918de2fa581c585
Oct 30 21:25:55.525464 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:55.525444    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"]
Oct 30 21:25:56.116018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:56.115927    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x" event={"ID":"70841359-02bc-4f62-a78c-fee5cc05a2f1","Type":"ContainerStarted","Data":"dff56640367a5f408bdf63da5fd2e043717076f03c735cfd5a19a98316b8f9bc"}
Oct 30 21:25:56.116018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:56.115988    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x" event={"ID":"70841359-02bc-4f62-a78c-fee5cc05a2f1","Type":"ContainerStarted","Data":"9d8b670089d27a3d0edc29af9f866b6179cb02f1d9f0529a3918de2fa581c585"}
Oct 30 21:25:56.136367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:56.136319    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x" podStartSLOduration=2.136304701 podStartE2EDuration="2.136304701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:56.134936796 +0000 UTC m=+2922.626753342" watchObservedRunningTime="2026-10-30 21:25:56.136304701 +0000 UTC m=+2922.628121245"
Oct 30 21:25:57.129711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:57.128680    4810 generic.go:356] "Generic (PLEG): container finished" podID="70841359-02bc-4f62-a78c-fee5cc05a2f1" containerID="dff56640367a5f408bdf63da5fd2e043717076f03c735cfd5a19a98316b8f9bc" exitCode=0
Oct 30 21:25:57.129711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:57.128726    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x" event={"ID":"70841359-02bc-4f62-a78c-fee5cc05a2f1","Type":"ContainerDied","Data":"dff56640367a5f408bdf63da5fd2e043717076f03c735cfd5a19a98316b8f9bc"}
Oct 30 21:25:58.309679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:58.309650    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:58.410824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:58.410744    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70841359-02bc-4f62-a78c-fee5cc05a2f1-kube-api-access-l8bxk\" (UniqueName: \"kubernetes.io/projected/70841359-02bc-4f62-a78c-fee5cc05a2f1-kube-api-access-l8bxk\") pod \"70841359-02bc-4f62-a78c-fee5cc05a2f1\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") "
Oct 30 21:25:58.410824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:58.410785    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-xks-ca\") pod \"70841359-02bc-4f62-a78c-fee5cc05a2f1\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") "
Oct 30 21:25:58.411095 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:58.410966    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-openshift-ca\") pod \"70841359-02bc-4f62-a78c-fee5cc05a2f1\" (UID: \"70841359-02bc-4f62-a78c-fee5cc05a2f1\") "
Oct 30 21:25:58.411353 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:58.411329    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-openshift-ca" pod "70841359-02bc-4f62-a78c-fee5cc05a2f1" (UID: "70841359-02bc-4f62-a78c-fee5cc05a2f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:58.411597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:58.411581    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:58.412910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:58.412876    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-xks-ca" pod "70841359-02bc-4f62-a78c-fee5cc05a2f1" (UID: "70841359-02bc-4f62-a78c-fee5cc05a2f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:58.413046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:58.412968    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70841359-02bc-4f62-a78c-fee5cc05a2f1-kube-api-access-l8bxk" pod "70841359-02bc-4f62-a78c-fee5cc05a2f1" (UID: "70841359-02bc-4f62-a78c-fee5cc05a2f1"). InnerVolumeSpecName "kube-api-access-l8bxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:58.512635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:58.512604    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8bxk\" (UniqueName: \"kubernetes.io/projected/70841359-02bc-4f62-a78c-fee5cc05a2f1-kube-api-access-l8bxk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:58.512635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:58.512633    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70841359-02bc-4f62-a78c-fee5cc05a2f1-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:59.156391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.153818    4810 generic.go:356] "Generic (PLEG): container finished" podID="7e70ac49-535d-4ba4-af02-4f15b3e4ed5b" containerID="5c190ee7d3a948ae65fd41b6fbcb9fd0796ba304739be52a757dbe5c6b55b2c8" exitCode=0
Oct 30 21:25:59.156391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.153952    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm" event={"ID":"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b","Type":"ContainerDied","Data":"5c190ee7d3a948ae65fd41b6fbcb9fd0796ba304739be52a757dbe5c6b55b2c8"}
Oct 30 21:25:59.156391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.156107    4810 generic.go:356] "Generic (PLEG): container finished" podID="e4f5604e-38c2-4f39-9f51-496b1a383cc9" containerID="b660eaaf20d9eff2dd70c643e0409caee011b9da0275a2a2dac2836b62178cc1" exitCode=0
Oct 30 21:25:59.156391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.156176    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l" event={"ID":"e4f5604e-38c2-4f39-9f51-496b1a383cc9","Type":"ContainerDied","Data":"b660eaaf20d9eff2dd70c643e0409caee011b9da0275a2a2dac2836b62178cc1"}
Oct 30 21:25:59.158291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.158223    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x" event={"ID":"70841359-02bc-4f62-a78c-fee5cc05a2f1","Type":"ContainerDied","Data":"9d8b670089d27a3d0edc29af9f866b6179cb02f1d9f0529a3918de2fa581c585"}
Oct 30 21:25:59.158490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.158471    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d8b670089d27a3d0edc29af9f866b6179cb02f1d9f0529a3918de2fa581c585"
Oct 30 21:25:59.158651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.158638    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"
Oct 30 21:25:59.231357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.231324    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"]
Oct 30 21:25:59.318876 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:25:59.304961    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70841359_02bc_4f62_a78c_fee5cc05a2f1.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:25:59.524887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.505389    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:25:59.600365 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.600283    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-plugins-config-volume\") pod \"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b\" (UID: \"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b\") "
Oct 30 21:25:59.600725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.600705    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-kube-api-access-sfp47\" (UniqueName: \"kubernetes.io/projected/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-kube-api-access-sfp47\") pod \"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b\" (UID: \"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b\") "
Oct 30 21:25:59.603818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.603334    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-plugins-config-volume" pod "7e70ac49-535d-4ba4-af02-4f15b3e4ed5b" (UID: "7e70ac49-535d-4ba4-af02-4f15b3e4ed5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:59.614992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.613210    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:59.646851 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.646074    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-kube-api-access-sfp47" pod "7e70ac49-535d-4ba4-af02-4f15b3e4ed5b" (UID: "7e70ac49-535d-4ba4-af02-4f15b3e4ed5b"). InnerVolumeSpecName "kube-api-access-sfp47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:59.679393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.679372    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:25:59.714815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.713726    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f5604e-38c2-4f39-9f51-496b1a383cc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f5604e-38c2-4f39-9f51-496b1a383cc9-plugins-config-volume\") pod \"e4f5604e-38c2-4f39-9f51-496b1a383cc9\" (UID: \"e4f5604e-38c2-4f39-9f51-496b1a383cc9\") "
Oct 30 21:25:59.714815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.713795    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f5604e-38c2-4f39-9f51-496b1a383cc9-kube-api-access-57ss8\" (UniqueName: \"kubernetes.io/projected/e4f5604e-38c2-4f39-9f51-496b1a383cc9-kube-api-access-57ss8\") pod \"e4f5604e-38c2-4f39-9f51-496b1a383cc9\" (UID: \"e4f5604e-38c2-4f39-9f51-496b1a383cc9\") "
Oct 30 21:25:59.714815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.714359    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f5604e-38c2-4f39-9f51-496b1a383cc9-plugins-config-volume" pod "e4f5604e-38c2-4f39-9f51-496b1a383cc9" (UID: "e4f5604e-38c2-4f39-9f51-496b1a383cc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:59.725013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.716690    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f5604e-38c2-4f39-9f51-496b1a383cc9-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:59.725013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.716719    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfp47\" (UniqueName: \"kubernetes.io/projected/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b-kube-api-access-sfp47\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:59.740084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.740046    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f5604e-38c2-4f39-9f51-496b1a383cc9-kube-api-access-57ss8" pod "e4f5604e-38c2-4f39-9f51-496b1a383cc9" (UID: "e4f5604e-38c2-4f39-9f51-496b1a383cc9"). InnerVolumeSpecName "kube-api-access-57ss8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:59.754332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.754296    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"]
Oct 30 21:25:59.820237 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.818811    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57ss8\" (UniqueName: \"kubernetes.io/projected/e4f5604e-38c2-4f39-9f51-496b1a383cc9-kube-api-access-57ss8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:59.899559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.899522    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"]
Oct 30 21:25:59.899787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:25:59.899742    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld" podUID="6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48" containerName="sim" containerID="cri-o://25717b2164ae41385f2c19647057ae19015398b12f8a57ecae5c6ff5fa9b5b09" gracePeriod=30
Oct 30 21:26:00.287906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.287851    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm" event={"ID":"7e70ac49-535d-4ba4-af02-4f15b3e4ed5b","Type":"ContainerDied","Data":"9b21cb3f097d31629af0c84bf48e86e53b51e67f112eabf47595607e3855628e"}
Oct 30 21:26:00.288043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.287971    4810 scope.go:122] "RemoveContainer" containerID="5c190ee7d3a948ae65fd41b6fbcb9fd0796ba304739be52a757dbe5c6b55b2c8"
Oct 30 21:26:00.288181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.288162    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"
Oct 30 21:26:00.293642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.291120    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"
Oct 30 21:26:00.293642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.291304    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l" event={"ID":"e4f5604e-38c2-4f39-9f51-496b1a383cc9","Type":"ContainerDied","Data":"0e9dcae8e3249294e2474e803a9400da91f194bbc6f0dea75ed37d4b8a4f88bd"}
Oct 30 21:26:00.319905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.317150    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"]
Oct 30 21:26:00.319905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.318038    4810 scope.go:122] "RemoveContainer" containerID="b660eaaf20d9eff2dd70c643e0409caee011b9da0275a2a2dac2836b62178cc1"
Oct 30 21:26:00.322153 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.322067    4810 generic.go:356] "Generic (PLEG): container finished" podID="6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48" containerID="25717b2164ae41385f2c19647057ae19015398b12f8a57ecae5c6ff5fa9b5b09" exitCode=0
Oct 30 21:26:00.322153 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.322106    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld" event={"ID":"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48","Type":"ContainerDied","Data":"25717b2164ae41385f2c19647057ae19015398b12f8a57ecae5c6ff5fa9b5b09"}
Oct 30 21:26:00.326933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.326905    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-74f9f4-66f7cbbd6c-c79gm"]
Oct 30 21:26:00.351283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.351257    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"]
Oct 30 21:26:00.357524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.357500    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-74f9f4-5db96dc479-v6c2l"]
Oct 30 21:26:00.506227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.506199    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:26:00.557837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.557769    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-kube-api-access-5r9mx\" (UniqueName: \"kubernetes.io/projected/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-kube-api-access-5r9mx\") pod \"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48\" (UID: \"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48\") "
Oct 30 21:26:00.558049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.557992    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-tls\" (UniqueName: \"kubernetes.io/secret/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-tls\") pod \"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48\" (UID: \"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48\") "
Oct 30 21:26:00.560646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.560620    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-tls" pod "6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48" (UID: "6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:00.561179 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.561154    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-kube-api-access-5r9mx" pod "6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48" (UID: "6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48"). InnerVolumeSpecName "kube-api-access-5r9mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:00.660433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.660402    4810 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-tls\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.660433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:00.660431    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r9mx\" (UniqueName: \"kubernetes.io/projected/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48-kube-api-access-5r9mx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:01.334201 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:01.334170    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"
Oct 30 21:26:01.334645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:01.334175    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld" event={"ID":"6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48","Type":"ContainerDied","Data":"2df11d47c91ae6af65356cfccdfa91bcd7e1d978c880794515d8694c4ceaff3d"}
Oct 30 21:26:01.334645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:01.334297    4810 scope.go:122] "RemoveContainer" containerID="25717b2164ae41385f2c19647057ae19015398b12f8a57ecae5c6ff5fa9b5b09"
Oct 30 21:26:01.360503 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:01.360476    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"]
Oct 30 21:26:01.364763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:01.364741    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-9a57c4/e2e-usage-sim-8fece6-85bfb989b-2dbld"]
Oct 30 21:26:01.664751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:01.664712    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48" path="/var/lib/kubelet/pods/6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48/volumes"
Oct 30 21:26:01.665337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:01.665312    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e70ac49-535d-4ba4-af02-4f15b3e4ed5b" path="/var/lib/kubelet/pods/7e70ac49-535d-4ba4-af02-4f15b3e4ed5b/volumes"
Oct 30 21:26:01.665763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:01.665749    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f5604e-38c2-4f39-9f51-496b1a383cc9" path="/var/lib/kubelet/pods/e4f5604e-38c2-4f39-9f51-496b1a383cc9/volumes"
Oct 30 21:26:04.231761 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.231717    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg" podUID="cc8065b8-f58e-42c0-8d6d-8718fee7446c" containerName="payload-processing" containerID="cri-o://d701dc2bc30306ade2ba2af98a39a3b8598662c2324e800904fae9c6852bf54f" gracePeriod=25
Oct 30 21:26:04.387628 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.387461    4810 generic.go:356] "Generic (PLEG): container finished" podID="cc8065b8-f58e-42c0-8d6d-8718fee7446c" containerID="d701dc2bc30306ade2ba2af98a39a3b8598662c2324e800904fae9c6852bf54f" exitCode=0
Oct 30 21:26:04.387628 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.387548    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg" event={"ID":"cc8065b8-f58e-42c0-8d6d-8718fee7446c","Type":"ContainerDied","Data":"d701dc2bc30306ade2ba2af98a39a3b8598662c2324e800904fae9c6852bf54f"}
Oct 30 21:26:04.534937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.534907    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:26:04.626648 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.626611    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-routing-overlay-volume\") pod \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") "
Oct 30 21:26:04.626648 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.626652    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-kube-api-access-7qzf9\" (UniqueName: \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-kube-api-access-7qzf9\") pod \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") "
Oct 30 21:26:04.627039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.627018    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-provider-credentials\") pod \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") "
Oct 30 21:26:04.627039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.627025    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-routing-overlay-volume" pod "cc8065b8-f58e-42c0-8d6d-8718fee7446c" (UID: "cc8065b8-f58e-42c0-8d6d-8718fee7446c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:04.627135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.627077    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-plugins-config-volume\") pod \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\" (UID: \"cc8065b8-f58e-42c0-8d6d-8718fee7446c\") "
Oct 30 21:26:04.627588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.627563    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-plugins-config-volume" pod "cc8065b8-f58e-42c0-8d6d-8718fee7446c" (UID: "cc8065b8-f58e-42c0-8d6d-8718fee7446c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:04.627700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.627660    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:04.627700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.627673    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc8065b8-f58e-42c0-8d6d-8718fee7446c-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:04.629024 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.628998    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-kube-api-access-7qzf9" pod "cc8065b8-f58e-42c0-8d6d-8718fee7446c" (UID: "cc8065b8-f58e-42c0-8d6d-8718fee7446c"). InnerVolumeSpecName "kube-api-access-7qzf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:04.629183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.629160    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-provider-credentials" pod "cc8065b8-f58e-42c0-8d6d-8718fee7446c" (UID: "cc8065b8-f58e-42c0-8d6d-8718fee7446c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:04.728828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.728794    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:04.728828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.728826    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qzf9\" (UniqueName: \"kubernetes.io/projected/cc8065b8-f58e-42c0-8d6d-8718fee7446c-kube-api-access-7qzf9\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:04.755217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:04.755143    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5" podUID="1388555d-21ed-4e4f-a201-307499a352dc" containerName="payload-processing" containerID="cri-o://1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600" gracePeriod=25
Oct 30 21:26:05.029731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.029702    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:26:05.134315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.134283    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-plugins-config-volume\") pod \"1388555d-21ed-4e4f-a201-307499a352dc\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") "
Oct 30 21:26:05.134522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.134379    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-routing-overlay-volume\") pod \"1388555d-21ed-4e4f-a201-307499a352dc\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") "
Oct 30 21:26:05.134522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.134504    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-kube-api-access-pzhn4\" (UniqueName: \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-kube-api-access-pzhn4\") pod \"1388555d-21ed-4e4f-a201-307499a352dc\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") "
Oct 30 21:26:05.134663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.134537    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-provider-credentials\") pod \"1388555d-21ed-4e4f-a201-307499a352dc\" (UID: \"1388555d-21ed-4e4f-a201-307499a352dc\") "
Oct 30 21:26:05.134852 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.134825    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-plugins-config-volume" pod "1388555d-21ed-4e4f-a201-307499a352dc" (UID: "1388555d-21ed-4e4f-a201-307499a352dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:05.134953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.134843    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-routing-overlay-volume" pod "1388555d-21ed-4e4f-a201-307499a352dc" (UID: "1388555d-21ed-4e4f-a201-307499a352dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:05.135376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.135351    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:05.135376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.135375    4810 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1388555d-21ed-4e4f-a201-307499a352dc-routing-overlay-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:05.136674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.136651    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-kube-api-access-pzhn4" pod "1388555d-21ed-4e4f-a201-307499a352dc" (UID: "1388555d-21ed-4e4f-a201-307499a352dc"). InnerVolumeSpecName "kube-api-access-pzhn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:05.136774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.136750    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-provider-credentials" pod "1388555d-21ed-4e4f-a201-307499a352dc" (UID: "1388555d-21ed-4e4f-a201-307499a352dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:05.237294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.237263    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzhn4\" (UniqueName: \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-kube-api-access-pzhn4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:05.237294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.237291    4810 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1388555d-21ed-4e4f-a201-307499a352dc-provider-credentials\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:05.418881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.414133    4810 generic.go:356] "Generic (PLEG): container finished" podID="1388555d-21ed-4e4f-a201-307499a352dc" containerID="1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600" exitCode=0
Oct 30 21:26:05.418881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.414409    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5" event={"ID":"1388555d-21ed-4e4f-a201-307499a352dc","Type":"ContainerDied","Data":"1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600"}
Oct 30 21:26:05.418881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.414459    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5" event={"ID":"1388555d-21ed-4e4f-a201-307499a352dc","Type":"ContainerDied","Data":"4aa4f5724913fe673b7dc047ec0ac4f926471c0c4a051c7d09bb208fe819bfee"}
Oct 30 21:26:05.418881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.414484    4810 scope.go:122] "RemoveContainer" containerID="1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600"
Oct 30 21:26:05.418881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.414598    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"
Oct 30 21:26:05.429699 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.421995    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg" event={"ID":"cc8065b8-f58e-42c0-8d6d-8718fee7446c","Type":"ContainerDied","Data":"c53dcf45d66a73ced148908fc1c139b748bdc3fd566ecfdba75d68c94fdf4fcc"}
Oct 30 21:26:05.431140 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.431110    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"
Oct 30 21:26:05.458305 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.458258    4810 scope.go:122] "RemoveContainer" containerID="1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600"
Oct 30 21:26:05.459685 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:26:05.459542    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600\": container with ID starting with 1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600 not found: ID does not exist" containerID="1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600"
Oct 30 21:26:05.459685 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.459582    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600"} err="failed to get container status \"1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600\": rpc error: code = NotFound desc = could not find container \"1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600\": container with ID starting with 1f9d6a454ee6796c292d5aa84e2f00cace06778f91688d57c9c17167a44a1600 not found: ID does not exist"
Oct 30 21:26:05.459685 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.459606    4810 scope.go:122] "RemoveContainer" containerID="d701dc2bc30306ade2ba2af98a39a3b8598662c2324e800904fae9c6852bf54f"
Oct 30 21:26:05.512311 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.512253    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"]
Oct 30 21:26:05.518291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.518269    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6741b1/payload-processing-external-model-e2e-extmod-a-6741b1-85d6lnmmg"]
Oct 30 21:26:05.540939 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.540912    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"]
Oct 30 21:26:05.546873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.546837    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d95261/payload-processing-external-model-e2e-extmod-b-d95261-5f87vsbd5"]
Oct 30 21:26:05.664340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.664304    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1388555d-21ed-4e4f-a201-307499a352dc" path="/var/lib/kubelet/pods/1388555d-21ed-4e4f-a201-307499a352dc/volumes"
Oct 30 21:26:05.664742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:05.664727    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8065b8-f58e-42c0-8d6d-8718fee7446c" path="/var/lib/kubelet/pods/cc8065b8-f58e-42c0-8d6d-8718fee7446c/volumes"
Oct 30 21:26:11.729047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:11.729011    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:26:18.917916 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:18.917882    4810 scope.go:122] "RemoveContainer" containerID="ef431eb5a73cbf885509b67dd30a6035a4627250b6ad80c0a8d3e3b84bc027f2"
Oct 30 21:26:18.929548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:18.929520    4810 scope.go:122] "RemoveContainer" containerID="f16f742d11571c7c9fd15132ada05bbf7dfc96167f5c07fec731904aebde52b2"
Oct 30 21:26:25.156721 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:26:25.156685    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:26:26.213519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.213471    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"]
Oct 30 21:26:26.215485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215455    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1388555d-21ed-4e4f-a201-307499a352dc" containerName="payload-processing"
Oct 30 21:26:26.215485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215484    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1388555d-21ed-4e4f-a201-307499a352dc" containerName="payload-processing"
Oct 30 21:26:26.215675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215571    4810 cpu_manager.go:414] "containerMap: removing container" podUID="70841359-02bc-4f62-a78c-fee5cc05a2f1" containerName="revoke-keys"
Oct 30 21:26:26.215675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215580    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70841359-02bc-4f62-a78c-fee5cc05a2f1" containerName="revoke-keys"
Oct 30 21:26:26.215675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215617    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e4f5604e-38c2-4f39-9f51-496b1a383cc9" containerName="payload-pre-processing"
Oct 30 21:26:26.215675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215625    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f5604e-38c2-4f39-9f51-496b1a383cc9" containerName="payload-pre-processing"
Oct 30 21:26:26.215675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215666    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48" containerName="sim"
Oct 30 21:26:26.215675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215674    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48" containerName="sim"
Oct 30 21:26:26.216138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215707    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cc8065b8-f58e-42c0-8d6d-8718fee7446c" containerName="payload-processing"
Oct 30 21:26:26.216138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215720    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8065b8-f58e-42c0-8d6d-8718fee7446c" containerName="payload-processing"
Oct 30 21:26:26.216138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215751    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7e70ac49-535d-4ba4-af02-4f15b3e4ed5b" containerName="payload-processing"
Oct 30 21:26:26.216138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.215758    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e70ac49-535d-4ba4-af02-4f15b3e4ed5b" containerName="payload-processing"
Oct 30 21:26:26.216354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.216199    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="70841359-02bc-4f62-a78c-fee5cc05a2f1" containerName="revoke-keys"
Oct 30 21:26:26.216354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.216234    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f5604e-38c2-4f39-9f51-496b1a383cc9" containerName="payload-pre-processing"
Oct 30 21:26:26.216354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.216265    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cdebae3-7b99-4d7b-b6bb-c6a30fd53e48" containerName="sim"
Oct 30 21:26:26.216354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.216288    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8065b8-f58e-42c0-8d6d-8718fee7446c" containerName="payload-processing"
Oct 30 21:26:26.216354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.216306    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e70ac49-535d-4ba4-af02-4f15b3e4ed5b" containerName="payload-processing"
Oct 30 21:26:26.216354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.216332    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1388555d-21ed-4e4f-a201-307499a352dc" containerName="payload-processing"
Oct 30 21:26:26.231817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.224685    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.231817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.227204    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"]
Oct 30 21:26:26.233675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.233537    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:26:26.233675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.233629    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:26.237287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.237264    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.243315 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.243291    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"]
Oct 30 21:26:26.251585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.251555    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"]
Oct 30 21:26:26.362171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.362141    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa325ed3-7w5gz\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.362342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.362254    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f513ec-sjt6h\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.362342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.362290    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f513ec-sjt6h\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.362438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.362376    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa325ed3-7w5gz\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.362438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.362421    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4p8\" (UniqueName: \"kubernetes.io/projected/f769b235-7234-4aee-9a42-3a49abcb6594-kube-api-access-dp4p8\") pod \"maas-api-revoke-sub-65f513ec-sjt6h\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.362533 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.362470    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4n5l\" (UniqueName: \"kubernetes.io/projected/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-kube-api-access-v4n5l\") pod \"maas-api-revoke-sub-fa325ed3-7w5gz\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.464411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.464335    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa325ed3-7w5gz\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.464635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.464426    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f513ec-sjt6h\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.464635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.464463    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f513ec-sjt6h\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.464635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.464498    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa325ed3-7w5gz\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.464910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.464837    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4p8\" (UniqueName: \"kubernetes.io/projected/f769b235-7234-4aee-9a42-3a49abcb6594-kube-api-access-dp4p8\") pod \"maas-api-revoke-sub-65f513ec-sjt6h\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.465040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.464983    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4n5l\" (UniqueName: \"kubernetes.io/projected/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-kube-api-access-v4n5l\") pod \"maas-api-revoke-sub-fa325ed3-7w5gz\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.465321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.465169    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa325ed3-7w5gz\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.465555 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.465528    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f513ec-sjt6h\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.467542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.467515    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa325ed3-7w5gz\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.467758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.467740    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f513ec-sjt6h\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.473958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.473934    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4n5l\" (UniqueName: \"kubernetes.io/projected/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-kube-api-access-v4n5l\") pod \"maas-api-revoke-sub-fa325ed3-7w5gz\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.474412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.474385    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4p8\" (UniqueName: \"kubernetes.io/projected/f769b235-7234-4aee-9a42-3a49abcb6594-kube-api-access-dp4p8\") pod \"maas-api-revoke-sub-65f513ec-sjt6h\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.552957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.552924    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:26.566137 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.566107    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:26.858339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.858309    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"]
Oct 30 21:26:26.859596 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:26:26.859571    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf769b235_7234_4aee_9a42_3a49abcb6594.slice/crio-51d024d3610aa32a1ea7df7b295fa82bc1257d93794a5446f82e951123634283 WatchSource:0}: Error finding container 51d024d3610aa32a1ea7df7b295fa82bc1257d93794a5446f82e951123634283: Status 404 returned error can't find the container with id 51d024d3610aa32a1ea7df7b295fa82bc1257d93794a5446f82e951123634283
Oct 30 21:26:26.881446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:26.881420    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"]
Oct 30 21:26:26.883110 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:26:26.883082    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dce06c_45c2_476f_aa9d_ca0bfe42c86d.slice/crio-e44f59a16a909a22d3b9defed02a944e3d380bd74093797b4bb5376233a2633c WatchSource:0}: Error finding container e44f59a16a909a22d3b9defed02a944e3d380bd74093797b4bb5376233a2633c: Status 404 returned error can't find the container with id e44f59a16a909a22d3b9defed02a944e3d380bd74093797b4bb5376233a2633c
Oct 30 21:26:27.738664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:27.738632    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz" event={"ID":"56dce06c-45c2-476f-aa9d-ca0bfe42c86d","Type":"ContainerStarted","Data":"7d53807d4ee7e0030bd10bef5b00b7a3e323d365787c3235bc0788c2462553d6"}
Oct 30 21:26:27.739136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:27.738686    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz" event={"ID":"56dce06c-45c2-476f-aa9d-ca0bfe42c86d","Type":"ContainerStarted","Data":"e44f59a16a909a22d3b9defed02a944e3d380bd74093797b4bb5376233a2633c"}
Oct 30 21:26:27.740345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:27.740315    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h" event={"ID":"f769b235-7234-4aee-9a42-3a49abcb6594","Type":"ContainerStarted","Data":"60f78cec43ffbccaf91c4344d58bcb62e5b35fd1f8dfd1e47617e57e31222280"}
Oct 30 21:26:27.740452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:27.740365    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h" event={"ID":"f769b235-7234-4aee-9a42-3a49abcb6594","Type":"ContainerStarted","Data":"51d024d3610aa32a1ea7df7b295fa82bc1257d93794a5446f82e951123634283"}
Oct 30 21:26:27.785584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:27.785537    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz" podStartSLOduration=1.785522799 podStartE2EDuration="1.785522799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:27.756236366 +0000 UTC m=+2954.248052912" watchObservedRunningTime="2026-10-30 21:26:27.785522799 +0000 UTC m=+2954.277339369"
Oct 30 21:26:27.799944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:27.799890    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h" podStartSLOduration=1.799871926 podStartE2EDuration="1.799871926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:27.796185326 +0000 UTC m=+2954.288001872" watchObservedRunningTime="2026-10-30 21:26:27.799871926 +0000 UTC m=+2954.291688462"
Oct 30 21:26:28.763401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:28.763361    4810 generic.go:356] "Generic (PLEG): container finished" podID="56dce06c-45c2-476f-aa9d-ca0bfe42c86d" containerID="7d53807d4ee7e0030bd10bef5b00b7a3e323d365787c3235bc0788c2462553d6" exitCode=0
Oct 30 21:26:28.763899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:28.763446    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz" event={"ID":"56dce06c-45c2-476f-aa9d-ca0bfe42c86d","Type":"ContainerDied","Data":"7d53807d4ee7e0030bd10bef5b00b7a3e323d365787c3235bc0788c2462553d6"}
Oct 30 21:26:28.765779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:28.765750    4810 generic.go:356] "Generic (PLEG): container finished" podID="f769b235-7234-4aee-9a42-3a49abcb6594" containerID="60f78cec43ffbccaf91c4344d58bcb62e5b35fd1f8dfd1e47617e57e31222280" exitCode=0
Oct 30 21:26:28.765958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:28.765828    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h" event={"ID":"f769b235-7234-4aee-9a42-3a49abcb6594","Type":"ContainerDied","Data":"60f78cec43ffbccaf91c4344d58bcb62e5b35fd1f8dfd1e47617e57e31222280"}
Oct 30 21:26:29.967905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:29.967874    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:30.021573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.021544    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-openshift-ca\") pod \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") "
Oct 30 21:26:30.021785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.021759    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-xks-ca\") pod \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") "
Oct 30 21:26:30.022111 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.021945    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-openshift-ca" pod "56dce06c-45c2-476f-aa9d-ca0bfe42c86d" (UID: "56dce06c-45c2-476f-aa9d-ca0bfe42c86d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:30.022111 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.022034    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-kube-api-access-v4n5l\" (UniqueName: \"kubernetes.io/projected/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-kube-api-access-v4n5l\") pod \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\" (UID: \"56dce06c-45c2-476f-aa9d-ca0bfe42c86d\") "
Oct 30 21:26:30.022599 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.022574    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:30.024617 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.024594    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-xks-ca" pod "56dce06c-45c2-476f-aa9d-ca0bfe42c86d" (UID: "56dce06c-45c2-476f-aa9d-ca0bfe42c86d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:30.024886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.024831    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-kube-api-access-v4n5l" pod "56dce06c-45c2-476f-aa9d-ca0bfe42c86d" (UID: "56dce06c-45c2-476f-aa9d-ca0bfe42c86d"). InnerVolumeSpecName "kube-api-access-v4n5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:30.045117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.045098    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:30.123637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.123606    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f769b235-7234-4aee-9a42-3a49abcb6594-kube-api-access-dp4p8\" (UniqueName: \"kubernetes.io/projected/f769b235-7234-4aee-9a42-3a49abcb6594-kube-api-access-dp4p8\") pod \"f769b235-7234-4aee-9a42-3a49abcb6594\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") "
Oct 30 21:26:30.123816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.123701    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-openshift-ca\") pod \"f769b235-7234-4aee-9a42-3a49abcb6594\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") "
Oct 30 21:26:30.123980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.123959    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-xks-ca\") pod \"f769b235-7234-4aee-9a42-3a49abcb6594\" (UID: \"f769b235-7234-4aee-9a42-3a49abcb6594\") "
Oct 30 21:26:30.124126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.124105    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-openshift-ca" pod "f769b235-7234-4aee-9a42-3a49abcb6594" (UID: "f769b235-7234-4aee-9a42-3a49abcb6594"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:30.124542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.124522    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:30.124637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.124544    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4n5l\" (UniqueName: \"kubernetes.io/projected/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-kube-api-access-v4n5l\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:30.124637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.124559    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56dce06c-45c2-476f-aa9d-ca0bfe42c86d-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:30.125731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.125706    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f769b235-7234-4aee-9a42-3a49abcb6594-kube-api-access-dp4p8" pod "f769b235-7234-4aee-9a42-3a49abcb6594" (UID: "f769b235-7234-4aee-9a42-3a49abcb6594"). InnerVolumeSpecName "kube-api-access-dp4p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:30.126072 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.126054    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-xks-ca" pod "f769b235-7234-4aee-9a42-3a49abcb6594" (UID: "f769b235-7234-4aee-9a42-3a49abcb6594"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:30.226630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.226598    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f769b235-7234-4aee-9a42-3a49abcb6594-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:30.226759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.226635    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4p8\" (UniqueName: \"kubernetes.io/projected/f769b235-7234-4aee-9a42-3a49abcb6594-kube-api-access-dp4p8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:30.786930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.786888    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz" event={"ID":"56dce06c-45c2-476f-aa9d-ca0bfe42c86d","Type":"ContainerDied","Data":"e44f59a16a909a22d3b9defed02a944e3d380bd74093797b4bb5376233a2633c"}
Oct 30 21:26:30.787119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.786956    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e44f59a16a909a22d3b9defed02a944e3d380bd74093797b4bb5376233a2633c"
Oct 30 21:26:30.787119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.786908    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"
Oct 30 21:26:30.788674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.788652    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"
Oct 30 21:26:30.788933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.788676    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h" event={"ID":"f769b235-7234-4aee-9a42-3a49abcb6594","Type":"ContainerDied","Data":"51d024d3610aa32a1ea7df7b295fa82bc1257d93794a5446f82e951123634283"}
Oct 30 21:26:30.789087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:30.788960    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51d024d3610aa32a1ea7df7b295fa82bc1257d93794a5446f82e951123634283"
Oct 30 21:26:53.647766 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:53.647731    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"]
Oct 30 21:26:53.651961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:53.651939    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8e4acb-psn9x"]
Oct 30 21:26:53.664515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:53.664489    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70841359-02bc-4f62-a78c-fee5cc05a2f1" path="/var/lib/kubelet/pods/70841359-02bc-4f62-a78c-fee5cc05a2f1/volumes"
Oct 30 21:26:54.032761 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:54.032682    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"]
Oct 30 21:26:54.037061 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:54.037039    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8f8744-t7vqv"]
Oct 30 21:26:55.157427 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:26:55.157394    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:26:55.664583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:26:55.664552    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a994b6bb-cb12-42e7-b541-e8e4440fbbd8" path="/var/lib/kubelet/pods/a994b6bb-cb12-42e7-b541-e8e4440fbbd8/volumes"
Oct 30 21:27:01.831143 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:01.831096    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"]
Oct 30 21:27:01.834634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:01.834581    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy" containerID="cri-o://3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f" gracePeriod=30
Oct 30 21:27:07.161890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.160060    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 21:27:07.214082 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.214051    4810 generic.go:356] "Generic (PLEG): container finished" podID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerID="3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f" exitCode=0
Oct 30 21:27:07.214453 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.214435    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"
Oct 30 21:27:07.214913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.214890    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" event={"ID":"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e","Type":"ContainerDied","Data":"3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f"}
Oct 30 21:27:07.215093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.215062    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj" event={"ID":"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e","Type":"ContainerDied","Data":"f163859a38365906fc868573a9b63d8370cb38dd94864f579ab1baa6304b1030"}
Oct 30 21:27:07.215190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.215098    4810 scope.go:122] "RemoveContainer" containerID="3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f"
Oct 30 21:27:07.236612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.236583    4810 scope.go:122] "RemoveContainer" containerID="3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f"
Oct 30 21:27:07.236976 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:27:07.236953    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f\": container with ID starting with 3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f not found: ID does not exist" containerID="3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f"
Oct 30 21:27:07.237104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.237000    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f"} err="failed to get container status \"3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f\": rpc error: code = NotFound desc = could not find container \"3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f\": container with ID starting with 3aea519b695731ff1c30d0275665801d00e05f3d1ffa9befc13e38044fc1e31f not found: ID does not exist"
Oct 30 21:27:07.254026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254003    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-credential-socket\") pod \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") "
Oct 30 21:27:07.254155 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254040    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-token\" (UniqueName: \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-token\") pod \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") "
Oct 30 21:27:07.254155 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254147    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-certs\") pod \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") "
Oct 30 21:27:07.254348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254330    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-data\") pod \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") "
Oct 30 21:27:07.254442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254426    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-envoy\") pod \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") "
Oct 30 21:27:07.254531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254459    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-podinfo\") pod \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") "
Oct 30 21:27:07.254531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254478    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-certs" pod "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" (UID: "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:07.254531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254503    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-socket\") pod \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") "
Oct 30 21:27:07.254695 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254578    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istiod-ca-cert\") pod \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") "
Oct 30 21:27:07.254695 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254631    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-kube-api-access-mtfgs\" (UniqueName: \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-kube-api-access-mtfgs\") pod \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\" (UID: \"a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e\") "
Oct 30 21:27:07.255451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254927    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-credential-socket" pod "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" (UID: "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:07.255451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.254949    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-data" pod "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" (UID: "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:07.255451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.255065    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-socket" pod "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" (UID: "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:07.255451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.255397    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istiod-ca-cert" pod "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" (UID: "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:07.255451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.255414    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:27:07.255451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.255436    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:27:07.255451 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.255451    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:27:07.255965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.255465    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:27:07.256681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.256645    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-token" pod "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" (UID: "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:07.257772 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.257747    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-envoy" pod "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" (UID: "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:27:07.259360 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.259338    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-podinfo" pod "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" (UID: "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:27:07.259522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.259348    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-kube-api-access-mtfgs" pod "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" (UID: "a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e"). InnerVolumeSpecName "kube-api-access-mtfgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:07.357062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.357022    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:27:07.357062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.357054    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:27:07.357062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.357068    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:27:07.357318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.357080    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfgs\" (UniqueName: \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-kube-api-access-mtfgs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:27:07.357318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.357093    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:27:07.546080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.546048    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"]
Oct 30 21:27:07.550304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.550273    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2wnxj"]
Oct 30 21:27:07.665787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:07.665759    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" path="/var/lib/kubelet/pods/a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e/volumes"
Oct 30 21:27:14.866960 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:14.866929    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:27:14.874762 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:14.874740    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:27:24.532206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:24.532166    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:27:25.161213 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:27:25.161175    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:27:51.286296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.284336    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"]
Oct 30 21:27:51.286296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.285352    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f769b235-7234-4aee-9a42-3a49abcb6594" containerName="revoke-keys"
Oct 30 21:27:51.286296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.285372    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f769b235-7234-4aee-9a42-3a49abcb6594" containerName="revoke-keys"
Oct 30 21:27:51.286296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.285397    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy"
Oct 30 21:27:51.286296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.285405    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy"
Oct 30 21:27:51.286296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.285430    4810 cpu_manager.go:414] "containerMap: removing container" podUID="56dce06c-45c2-476f-aa9d-ca0bfe42c86d" containerName="revoke-keys"
Oct 30 21:27:51.286296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.285438    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56dce06c-45c2-476f-aa9d-ca0bfe42c86d" containerName="revoke-keys"
Oct 30 21:27:51.286296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.285709    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f769b235-7234-4aee-9a42-3a49abcb6594" containerName="revoke-keys"
Oct 30 21:27:51.286296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.285731    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d6c287-ec16-4df8-a8d0-c57aa9bbaa2e" containerName="istio-proxy"
Oct 30 21:27:51.286296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.285743    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="56dce06c-45c2-476f-aa9d-ca0bfe42c86d" containerName="revoke-keys"
Oct 30 21:27:51.309379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.309341    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.321529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.321500    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0c4052\"/\"default-dockercfg-8p68f\""
Oct 30 21:27:51.322052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.321884    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0c4052\"/\"kube-root-ca.crt\""
Oct 30 21:27:51.323165 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.321953    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0c4052\"/\"test-model-0c4052-kserve-self-signed-certs\""
Oct 30 21:27:51.323165 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.322005    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0c4052\"/\"openshift-service-ca.crt\""
Oct 30 21:27:51.331302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.331275    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"]
Oct 30 21:27:51.425364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.425332    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d65735-3af5-46d9-9bd5-b616c14ab940-tls-certs\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.425581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.425397    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-home\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.425581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.425510    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-dshm\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.425742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.425724    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-tmp-dir\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.425818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.425807    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkj5\" (UniqueName: \"kubernetes.io/projected/87d65735-3af5-46d9-9bd5-b616c14ab940-kube-api-access-5nkj5\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.425899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.425840    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-model-cache\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.539931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.532617    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-tmp-dir\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.539931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.532706    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkj5\" (UniqueName: \"kubernetes.io/projected/87d65735-3af5-46d9-9bd5-b616c14ab940-kube-api-access-5nkj5\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.539931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.532749    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-model-cache\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.539931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.532846    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d65735-3af5-46d9-9bd5-b616c14ab940-tls-certs\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.539931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.532918    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-home\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.539931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.532943    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-dshm\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.539931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.535459    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-model-cache\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.539931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.535734    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-tmp-dir\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.539931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.536746    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-home\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.568343 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.568309    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkj5\" (UniqueName: \"kubernetes.io/projected/87d65735-3af5-46d9-9bd5-b616c14ab940-kube-api-access-5nkj5\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.569918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.569891    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-dshm\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.580681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.580643    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d65735-3af5-46d9-9bd5-b616c14ab940-tls-certs\") pod \"test-model-0c4052-kserve-6bd47968dc-7bgrm\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") " pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.639057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.639023    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:51.952921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:51.952889    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"]
Oct 30 21:27:51.953447 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:27:51.953419    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87d65735_3af5_46d9_9bd5_b616c14ab940.slice/crio-868a8ebfbbce424e9f58a25ce895b8eb7da6df4049dd918e5e04e080fff8d7eb WatchSource:0}: Error finding container 868a8ebfbbce424e9f58a25ce895b8eb7da6df4049dd918e5e04e080fff8d7eb: Status 404 returned error can't find the container with id 868a8ebfbbce424e9f58a25ce895b8eb7da6df4049dd918e5e04e080fff8d7eb
Oct 30 21:27:52.682402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:52.682363    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm" event={"ID":"87d65735-3af5-46d9-9bd5-b616c14ab940","Type":"ContainerStarted","Data":"b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c"}
Oct 30 21:27:52.682903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:52.682424    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm" event={"ID":"87d65735-3af5-46d9-9bd5-b616c14ab940","Type":"ContainerStarted","Data":"868a8ebfbbce424e9f58a25ce895b8eb7da6df4049dd918e5e04e080fff8d7eb"}
Oct 30 21:27:52.707495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:52.707436    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm" podStartSLOduration=1.707417891 podStartE2EDuration="1.707417891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:52.700836779 +0000 UTC m=+3039.192653329" watchObservedRunningTime="2026-10-30 21:27:52.707417891 +0000 UTC m=+3039.199234435"
Oct 30 21:27:53.639852 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:53.639813    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:27:55.025889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:55.025825    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"]
Oct 30 21:27:55.030124 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:55.030093    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a163532-dbdjw"]
Oct 30 21:27:55.158303 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:27:55.158268    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:27:55.672637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:55.672597    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4bbbba3-7b21-4172-8154-b930de12ad0f" path="/var/lib/kubelet/pods/a4bbbba3-7b21-4172-8154-b930de12ad0f/volumes"
Oct 30 21:27:58.480944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:58.480903    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:27:59.035827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:59.035791    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"]
Oct 30 21:27:59.040122 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:59.040093    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba5848c-ptfrj"]
Oct 30 21:27:59.694459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:27:59.694424    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f441d2-d449-4a47-9ec9-2af6280e89bb" path="/var/lib/kubelet/pods/11f441d2-d449-4a47-9ec9-2af6280e89bb/volumes"
Oct 30 21:28:01.639357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:01.639195    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:28:01.652087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:01.652057    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:28:01.781360 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:01.781329    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:28:02.647675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.647642    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"]
Oct 30 21:28:02.668405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.668373    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.680499 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.680471    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f8e2c3\"/\"kube-root-ca.crt\""
Oct 30 21:28:02.682350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.682177    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f8e2c3\"/\"default-dockercfg-nqf29\""
Oct 30 21:28:02.683711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.682600    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f8e2c3\"/\"openshift-service-ca.crt\""
Oct 30 21:28:02.683891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.682689    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f8e2c3\"/\"test-model-f8e2c3-kserve-self-signed-certs\""
Oct 30 21:28:02.695414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.695386    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"]
Oct 30 21:28:02.737094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.737057    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqcg8\" (UniqueName: \"kubernetes.io/projected/60648dda-a037-461c-92d4-c9e110f1e312-kube-api-access-vqcg8\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.737299 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.737221    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-model-cache\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.737299 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.737284    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-tmp-dir\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.737437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.737378    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-home\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.737437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.737417    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60648dda-a037-461c-92d4-c9e110f1e312-tls-certs\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.737984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.737953    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-dshm\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.839006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.838952    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-tmp-dir\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.839212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.839136    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-home\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.839212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.839181    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60648dda-a037-461c-92d4-c9e110f1e312-tls-certs\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.839333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.839252    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-dshm\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.839333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.839300    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqcg8\" (UniqueName: \"kubernetes.io/projected/60648dda-a037-461c-92d4-c9e110f1e312-kube-api-access-vqcg8\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.839458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.839437    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-model-cache\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.863922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.840800    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-tmp-dir\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.864191 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.841061    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-home\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.864335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.841270    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-model-cache\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.883558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.883520    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60648dda-a037-461c-92d4-c9e110f1e312-tls-certs\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.891245 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.884464    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqcg8\" (UniqueName: \"kubernetes.io/projected/60648dda-a037-461c-92d4-c9e110f1e312-kube-api-access-vqcg8\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.911923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.908579    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-dshm\") pod \"test-model-f8e2c3-kserve-955d688d9-5tlvc\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") " pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:02.991421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:02.991387    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:03.290479 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:28:03.290436    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60648dda_a037_461c_92d4_c9e110f1e312.slice/crio-5552c64c3bc86ac0dc682f19ec57a8c452b167f3fc6bade472a62582874c29fb WatchSource:0}: Error finding container 5552c64c3bc86ac0dc682f19ec57a8c452b167f3fc6bade472a62582874c29fb: Status 404 returned error can't find the container with id 5552c64c3bc86ac0dc682f19ec57a8c452b167f3fc6bade472a62582874c29fb
Oct 30 21:28:03.292040 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:03.292011    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"]
Oct 30 21:28:03.792674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:03.792635    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc" event={"ID":"60648dda-a037-461c-92d4-c9e110f1e312","Type":"ContainerStarted","Data":"d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717"}
Oct 30 21:28:03.793333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:03.793301    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc" event={"ID":"60648dda-a037-461c-92d4-c9e110f1e312","Type":"ContainerStarted","Data":"5552c64c3bc86ac0dc682f19ec57a8c452b167f3fc6bade472a62582874c29fb"}
Oct 30 21:28:03.810440 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:03.810399    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc" podStartSLOduration=1.810385852 podStartE2EDuration="1.810385852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:03.808727562 +0000 UTC m=+3050.300544108" watchObservedRunningTime="2026-10-30 21:28:03.810385852 +0000 UTC m=+3050.302202418"
Oct 30 21:28:03.991816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:03.991760    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:04.025126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:04.025094    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"]
Oct 30 21:28:04.029121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:04.029096    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159cec04-t29sq"]
Oct 30 21:28:05.032610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:05.032572    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"]
Oct 30 21:28:05.036636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:05.036613    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f29c044-77c2r"]
Oct 30 21:28:05.664479 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:05.664448    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d23be3-ef68-49b7-aa8b-6390aa3afec8" path="/var/lib/kubelet/pods/94d23be3-ef68-49b7-aa8b-6390aa3afec8/volumes"
Oct 30 21:28:05.664958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:05.664943    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b2f9f1-aeda-4825-8d00-74c511cce0c0" path="/var/lib/kubelet/pods/c9b2f9f1-aeda-4825-8d00-74c511cce0c0/volumes"
Oct 30 21:28:09.036414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:09.036379    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"]
Oct 30 21:28:09.040240 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:09.040214    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74ced40b-2glv9"]
Oct 30 21:28:09.664266 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:09.664231    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c38e4a-efb4-4d68-871e-a509a984d293" path="/var/lib/kubelet/pods/55c38e4a-efb4-4d68-871e-a509a984d293/volumes"
Oct 30 21:28:10.025647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:10.025561    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"]
Oct 30 21:28:10.029438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:10.029414    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa51bea-8zv4t"]
Oct 30 21:28:11.664430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:11.664399    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155a6d44-f3a1-4c4a-9f89-075655b8cb4c" path="/var/lib/kubelet/pods/155a6d44-f3a1-4c4a-9f89-075655b8cb4c/volumes"
Oct 30 21:28:12.991822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:12.991776    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:13.005565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:13.005538    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:13.888424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:13.888392    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:28:19.030337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:19.030301    4810 scope.go:122] "RemoveContainer" containerID="a5e6a92a57e50d9c44ea53ca33f46951e784b913be2a8a85e612efe38d377bf0"
Oct 30 21:28:22.031777 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:22.031738    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"]
Oct 30 21:28:22.038105 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:22.038074    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d407f0c1-bkgcq"]
Oct 30 21:28:23.666252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:23.666216    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede1ea4e-4f15-4e9f-92ca-ca70f9925dff" path="/var/lib/kubelet/pods/ede1ea4e-4f15-4e9f-92ca-ca70f9925dff/volumes"
Oct 30 21:28:25.159148 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:28:25.159111    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:28:34.039457 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:34.039274    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"]
Oct 30 21:28:34.048749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:34.048709    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-501a5638-sspns"]
Oct 30 21:28:35.665851 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:35.665817    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd34c707-cc17-4fc8-9222-16f625340ce3" path="/var/lib/kubelet/pods/dd34c707-cc17-4fc8-9222-16f625340ce3/volumes"
Oct 30 21:28:50.100006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:28:50.099966    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:28:55.158204 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:28:55.158122    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:29:15.256297 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.256259    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"]
Oct 30 21:29:15.261421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.261395    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.270157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.270128    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"]
Oct 30 21:29:15.276185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.276161    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:29:15.276467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.276439    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:15.397713 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.397660    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9108683c-sdbdm\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.398158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.398116    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncbkh\" (UniqueName: \"kubernetes.io/projected/8e20dea9-a72f-4811-a049-a8fcab2540ec-kube-api-access-ncbkh\") pod \"maas-api-revoke-sub-9108683c-sdbdm\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.398478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.398458    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9108683c-sdbdm\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.501974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.501926    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9108683c-sdbdm\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.502177 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.502025    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncbkh\" (UniqueName: \"kubernetes.io/projected/8e20dea9-a72f-4811-a049-a8fcab2540ec-kube-api-access-ncbkh\") pod \"maas-api-revoke-sub-9108683c-sdbdm\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.502177 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.502169    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9108683c-sdbdm\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.502902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.502878    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9108683c-sdbdm\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.504647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.504620    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9108683c-sdbdm\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.513907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.513826    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncbkh\" (UniqueName: \"kubernetes.io/projected/8e20dea9-a72f-4811-a049-a8fcab2540ec-kube-api-access-ncbkh\") pod \"maas-api-revoke-sub-9108683c-sdbdm\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.588276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.588243    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:15.877258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:15.877229    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"]
Oct 30 21:29:16.557293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:16.557170    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm" event={"ID":"8e20dea9-a72f-4811-a049-a8fcab2540ec","Type":"ContainerStarted","Data":"c4f08b6ab49c1c0d39573aa992ce9fc417a796ac97cb9e2642f2716b6a4e7b50"}
Oct 30 21:29:16.557293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:16.557237    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm" event={"ID":"8e20dea9-a72f-4811-a049-a8fcab2540ec","Type":"ContainerStarted","Data":"66cfa9435b1472baed46069e1210d8cfba80a474f1767fd73a99bb09c5f6798e"}
Oct 30 21:29:16.573577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:16.573521    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm" podStartSLOduration=1.573502773 podStartE2EDuration="1.573502773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:16.572152262 +0000 UTC m=+3123.063968807" watchObservedRunningTime="2026-10-30 21:29:16.573502773 +0000 UTC m=+3123.065319319"
Oct 30 21:29:17.582099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:17.582059    4810 generic.go:356] "Generic (PLEG): container finished" podID="8e20dea9-a72f-4811-a049-a8fcab2540ec" containerID="c4f08b6ab49c1c0d39573aa992ce9fc417a796ac97cb9e2642f2716b6a4e7b50" exitCode=0
Oct 30 21:29:17.582612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:17.582370    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm" event={"ID":"8e20dea9-a72f-4811-a049-a8fcab2540ec","Type":"ContainerDied","Data":"c4f08b6ab49c1c0d39573aa992ce9fc417a796ac97cb9e2642f2716b6a4e7b50"}
Oct 30 21:29:18.843482 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:18.843417    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:18.969236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:18.969204    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e20dea9-a72f-4811-a049-a8fcab2540ec-kube-api-access-ncbkh\" (UniqueName: \"kubernetes.io/projected/8e20dea9-a72f-4811-a049-a8fcab2540ec-kube-api-access-ncbkh\") pod \"8e20dea9-a72f-4811-a049-a8fcab2540ec\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") "
Oct 30 21:29:18.969444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:18.969344    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-openshift-ca\") pod \"8e20dea9-a72f-4811-a049-a8fcab2540ec\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") "
Oct 30 21:29:18.969525 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:18.969493    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-xks-ca\") pod \"8e20dea9-a72f-4811-a049-a8fcab2540ec\" (UID: \"8e20dea9-a72f-4811-a049-a8fcab2540ec\") "
Oct 30 21:29:18.969740 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:18.969712    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-openshift-ca" pod "8e20dea9-a72f-4811-a049-a8fcab2540ec" (UID: "8e20dea9-a72f-4811-a049-a8fcab2540ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:18.969995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:18.969968    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:18.971538 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:18.971510    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-xks-ca" pod "8e20dea9-a72f-4811-a049-a8fcab2540ec" (UID: "8e20dea9-a72f-4811-a049-a8fcab2540ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:18.971629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:18.971523    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e20dea9-a72f-4811-a049-a8fcab2540ec-kube-api-access-ncbkh" pod "8e20dea9-a72f-4811-a049-a8fcab2540ec" (UID: "8e20dea9-a72f-4811-a049-a8fcab2540ec"). InnerVolumeSpecName "kube-api-access-ncbkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:19.062664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.062632    4810 scope.go:122] "RemoveContainer" containerID="e4b7e31026d4e5af47af0ac7e977f4d8ee6f2d742f049fd799ae60638d42e82d"
Oct 30 21:29:19.071599 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.071564    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e20dea9-a72f-4811-a049-a8fcab2540ec-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:19.071599 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.071599    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncbkh\" (UniqueName: \"kubernetes.io/projected/8e20dea9-a72f-4811-a049-a8fcab2540ec-kube-api-access-ncbkh\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:19.074596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.074572    4810 scope.go:122] "RemoveContainer" containerID="4bc25b170215f65f7195536595b778f30e37ac0ff28ad5529f0f7dd20bd31431"
Oct 30 21:29:19.087611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.087591    4810 scope.go:122] "RemoveContainer" containerID="4ef88276733bd7be535172e7f69c01f2222bae86b217707213f85fa70e145b84"
Oct 30 21:29:19.099841 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.099817    4810 scope.go:122] "RemoveContainer" containerID="e348b20e3e83231a0263154de6a4b1d569c6bd6a40c5cc3c7f76e507f207e77b"
Oct 30 21:29:19.110767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.110747    4810 scope.go:122] "RemoveContainer" containerID="898039dfe63c3e643237e1b02d1a10e1bfd381845f5e7d7766053500e5d11901"
Oct 30 21:29:19.121712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.121683    4810 scope.go:122] "RemoveContainer" containerID="31c915baa0607fcaa22b6ea7d58f1b563b6505335aabd69910c3d0ca5bb9650d"
Oct 30 21:29:19.132572 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.132550    4810 scope.go:122] "RemoveContainer" containerID="3070172f53e0fb87457b8b719dd34d7ce42cd76d41df78d047d8da263486c450"
Oct 30 21:29:19.603645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.603612    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"
Oct 30 21:29:19.603645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.603643    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm" event={"ID":"8e20dea9-a72f-4811-a049-a8fcab2540ec","Type":"ContainerDied","Data":"66cfa9435b1472baed46069e1210d8cfba80a474f1767fd73a99bb09c5f6798e"}
Oct 30 21:29:19.603963 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:19.603693    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66cfa9435b1472baed46069e1210d8cfba80a474f1767fd73a99bb09c5f6798e"
Oct 30 21:29:25.156962 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:29:25.156923    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:29:25.889367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:25.889326    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"]
Oct 30 21:29:26.436929 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.435785    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"]
Oct 30 21:29:26.437431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.437141    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8e20dea9-a72f-4811-a049-a8fcab2540ec" containerName="revoke-keys"
Oct 30 21:29:26.437431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.437164    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e20dea9-a72f-4811-a049-a8fcab2540ec" containerName="revoke-keys"
Oct 30 21:29:26.437431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.437397    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e20dea9-a72f-4811-a049-a8fcab2540ec" containerName="revoke-keys"
Oct 30 21:29:26.446961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.444125    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"]
Oct 30 21:29:26.446961 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.444244    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:26.449715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.449693    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:26.450048 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.450023    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:29:26.526772 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.526731    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0acf2aa-47tpf\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:26.526772 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.526780    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2sb\" (UniqueName: \"kubernetes.io/projected/227cec8e-eed7-4036-84d4-7c3caabd5923-kube-api-access-7h2sb\") pod \"maas-api-revoke-sub-c0acf2aa-47tpf\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:26.527072 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.527045    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0acf2aa-47tpf\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:26.628824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.628779    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0acf2aa-47tpf\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:26.628824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.628821    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2sb\" (UniqueName: \"kubernetes.io/projected/227cec8e-eed7-4036-84d4-7c3caabd5923-kube-api-access-7h2sb\") pod \"maas-api-revoke-sub-c0acf2aa-47tpf\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:26.629178 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.629148    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0acf2aa-47tpf\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:26.629771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.629745    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0acf2aa-47tpf\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:26.632129 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.632101    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0acf2aa-47tpf\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:26.637371 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.637346    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2sb\" (UniqueName: \"kubernetes.io/projected/227cec8e-eed7-4036-84d4-7c3caabd5923-kube-api-access-7h2sb\") pod \"maas-api-revoke-sub-c0acf2aa-47tpf\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:26.773974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:26.773871    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:27.068831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:27.068801    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"]
Oct 30 21:29:27.069458 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:29:27.069436    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227cec8e_eed7_4036_84d4_7c3caabd5923.slice/crio-37900850f1d4c3b0810b4dc9eb6327174f6d1327f0363eb317f0293e61791160 WatchSource:0}: Error finding container 37900850f1d4c3b0810b4dc9eb6327174f6d1327f0363eb317f0293e61791160: Status 404 returned error can't find the container with id 37900850f1d4c3b0810b4dc9eb6327174f6d1327f0363eb317f0293e61791160
Oct 30 21:29:27.740989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:27.740887    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf" event={"ID":"227cec8e-eed7-4036-84d4-7c3caabd5923","Type":"ContainerStarted","Data":"02371637bd1b7ad02b401847ebddba93aa178a477115ad4d3c28524a6499d2cc"}
Oct 30 21:29:27.740989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:27.740978    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf" event={"ID":"227cec8e-eed7-4036-84d4-7c3caabd5923","Type":"ContainerStarted","Data":"37900850f1d4c3b0810b4dc9eb6327174f6d1327f0363eb317f0293e61791160"}
Oct 30 21:29:27.761207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:27.761148    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf" podStartSLOduration=1.761133546 podStartE2EDuration="1.761133546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:27.757510822 +0000 UTC m=+3134.249327367" watchObservedRunningTime="2026-10-30 21:29:27.761133546 +0000 UTC m=+3134.252950091"
Oct 30 21:29:28.627430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:28.627339    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:29:28.751360 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:28.751326    4810 generic.go:356] "Generic (PLEG): container finished" podID="227cec8e-eed7-4036-84d4-7c3caabd5923" containerID="02371637bd1b7ad02b401847ebddba93aa178a477115ad4d3c28524a6499d2cc" exitCode=0
Oct 30 21:29:28.751778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:28.751411    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf" event={"ID":"227cec8e-eed7-4036-84d4-7c3caabd5923","Type":"ContainerDied","Data":"02371637bd1b7ad02b401847ebddba93aa178a477115ad4d3c28524a6499d2cc"}
Oct 30 21:29:30.060553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.060511    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:30.143418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.143370    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227cec8e-eed7-4036-84d4-7c3caabd5923-kube-api-access-7h2sb\" (UniqueName: \"kubernetes.io/projected/227cec8e-eed7-4036-84d4-7c3caabd5923-kube-api-access-7h2sb\") pod \"227cec8e-eed7-4036-84d4-7c3caabd5923\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") "
Oct 30 21:29:30.143618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.143455    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-openshift-ca\") pod \"227cec8e-eed7-4036-84d4-7c3caabd5923\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") "
Oct 30 21:29:30.143618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.143591    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-xks-ca\") pod \"227cec8e-eed7-4036-84d4-7c3caabd5923\" (UID: \"227cec8e-eed7-4036-84d4-7c3caabd5923\") "
Oct 30 21:29:30.143852 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.143823    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-openshift-ca" pod "227cec8e-eed7-4036-84d4-7c3caabd5923" (UID: "227cec8e-eed7-4036-84d4-7c3caabd5923"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:30.144345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.144320    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:30.146480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.146453    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227cec8e-eed7-4036-84d4-7c3caabd5923-kube-api-access-7h2sb" pod "227cec8e-eed7-4036-84d4-7c3caabd5923" (UID: "227cec8e-eed7-4036-84d4-7c3caabd5923"). InnerVolumeSpecName "kube-api-access-7h2sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:30.149317 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.149295    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-xks-ca" pod "227cec8e-eed7-4036-84d4-7c3caabd5923" (UID: "227cec8e-eed7-4036-84d4-7c3caabd5923"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:30.245438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.245400    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2sb\" (UniqueName: \"kubernetes.io/projected/227cec8e-eed7-4036-84d4-7c3caabd5923-kube-api-access-7h2sb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:30.245438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.245430    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227cec8e-eed7-4036-84d4-7c3caabd5923-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:30.769745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.769705    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"
Oct 30 21:29:30.769745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.769719    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf" event={"ID":"227cec8e-eed7-4036-84d4-7c3caabd5923","Type":"ContainerDied","Data":"37900850f1d4c3b0810b4dc9eb6327174f6d1327f0363eb317f0293e61791160"}
Oct 30 21:29:30.770060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:30.769774    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37900850f1d4c3b0810b4dc9eb6327174f6d1327f0363eb317f0293e61791160"
Oct 30 21:29:32.229551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.229503    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"]
Oct 30 21:29:32.230910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.230887    4810 cpu_manager.go:414] "containerMap: removing container" podUID="227cec8e-eed7-4036-84d4-7c3caabd5923" containerName="revoke-keys"
Oct 30 21:29:32.230957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.230913    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227cec8e-eed7-4036-84d4-7c3caabd5923" containerName="revoke-keys"
Oct 30 21:29:32.231230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.231218    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="227cec8e-eed7-4036-84d4-7c3caabd5923" containerName="revoke-keys"
Oct 30 21:29:32.236414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.236389    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.240433 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.240402    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"]
Oct 30 21:29:32.242586 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.242560    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:32.242790 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.242773    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:29:32.288053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.288002    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46hl7\" (UniqueName: \"kubernetes.io/projected/a62e800f-d9a7-4539-84f6-ff97f5229123-kube-api-access-46hl7\") pod \"maas-api-revoke-sub-2bca2ab3-kkkjk\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.288260 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.288097    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bca2ab3-kkkjk\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.288375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.288344    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bca2ab3-kkkjk\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.389559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.389515    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bca2ab3-kkkjk\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.389775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.389609    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46hl7\" (UniqueName: \"kubernetes.io/projected/a62e800f-d9a7-4539-84f6-ff97f5229123-kube-api-access-46hl7\") pod \"maas-api-revoke-sub-2bca2ab3-kkkjk\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.389775 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.389655    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bca2ab3-kkkjk\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.390485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.390452    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bca2ab3-kkkjk\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.392642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.392617    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bca2ab3-kkkjk\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.398700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.398670    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46hl7\" (UniqueName: \"kubernetes.io/projected/a62e800f-d9a7-4539-84f6-ff97f5229123-kube-api-access-46hl7\") pod \"maas-api-revoke-sub-2bca2ab3-kkkjk\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.561745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.561658    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:32.859522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:32.859489    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"]
Oct 30 21:29:32.860138 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:29:32.860111    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62e800f_d9a7_4539_84f6_ff97f5229123.slice/crio-2033e9a01ad239c875a029b849d30ed9fce19a4159809894098b0c98256546a8 WatchSource:0}: Error finding container 2033e9a01ad239c875a029b849d30ed9fce19a4159809894098b0c98256546a8: Status 404 returned error can't find the container with id 2033e9a01ad239c875a029b849d30ed9fce19a4159809894098b0c98256546a8
Oct 30 21:29:33.805197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:33.805142    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk" event={"ID":"a62e800f-d9a7-4539-84f6-ff97f5229123","Type":"ContainerStarted","Data":"13e70e9532267f5e34ec54a60e906985153f3fb236dd5114844c4c1ebb099cb5"}
Oct 30 21:29:33.805697 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:33.805206    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk" event={"ID":"a62e800f-d9a7-4539-84f6-ff97f5229123","Type":"ContainerStarted","Data":"2033e9a01ad239c875a029b849d30ed9fce19a4159809894098b0c98256546a8"}
Oct 30 21:29:34.817747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:34.817709    4810 generic.go:356] "Generic (PLEG): container finished" podID="a62e800f-d9a7-4539-84f6-ff97f5229123" containerID="13e70e9532267f5e34ec54a60e906985153f3fb236dd5114844c4c1ebb099cb5" exitCode=0
Oct 30 21:29:34.818227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:34.817798    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk" event={"ID":"a62e800f-d9a7-4539-84f6-ff97f5229123","Type":"ContainerDied","Data":"13e70e9532267f5e34ec54a60e906985153f3fb236dd5114844c4c1ebb099cb5"}
Oct 30 21:29:35.014848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.014819    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:35.137967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.137935    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-xks-ca\") pod \"a62e800f-d9a7-4539-84f6-ff97f5229123\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") "
Oct 30 21:29:35.138193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.138055    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-openshift-ca\") pod \"a62e800f-d9a7-4539-84f6-ff97f5229123\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") "
Oct 30 21:29:35.138193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.138178    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62e800f-d9a7-4539-84f6-ff97f5229123-kube-api-access-46hl7\" (UniqueName: \"kubernetes.io/projected/a62e800f-d9a7-4539-84f6-ff97f5229123-kube-api-access-46hl7\") pod \"a62e800f-d9a7-4539-84f6-ff97f5229123\" (UID: \"a62e800f-d9a7-4539-84f6-ff97f5229123\") "
Oct 30 21:29:35.138429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.138394    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-openshift-ca" pod "a62e800f-d9a7-4539-84f6-ff97f5229123" (UID: "a62e800f-d9a7-4539-84f6-ff97f5229123"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:35.138746 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.138729    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:35.140230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.140203    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-xks-ca" pod "a62e800f-d9a7-4539-84f6-ff97f5229123" (UID: "a62e800f-d9a7-4539-84f6-ff97f5229123"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:35.140320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.140280    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62e800f-d9a7-4539-84f6-ff97f5229123-kube-api-access-46hl7" pod "a62e800f-d9a7-4539-84f6-ff97f5229123" (UID: "a62e800f-d9a7-4539-84f6-ff97f5229123"). InnerVolumeSpecName "kube-api-access-46hl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:35.243794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.241090    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46hl7\" (UniqueName: \"kubernetes.io/projected/a62e800f-d9a7-4539-84f6-ff97f5229123-kube-api-access-46hl7\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:35.251879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.251371    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62e800f-d9a7-4539-84f6-ff97f5229123-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:35.828556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.828523    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"
Oct 30 21:29:35.829057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.828522    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk" event={"ID":"a62e800f-d9a7-4539-84f6-ff97f5229123","Type":"ContainerDied","Data":"2033e9a01ad239c875a029b849d30ed9fce19a4159809894098b0c98256546a8"}
Oct 30 21:29:35.829057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:35.828670    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2033e9a01ad239c875a029b849d30ed9fce19a4159809894098b0c98256546a8"
Oct 30 21:29:37.021832 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.020794    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"]
Oct 30 21:29:37.287954 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.286570    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"]
Oct 30 21:29:37.290973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.288904    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a62e800f-d9a7-4539-84f6-ff97f5229123" containerName="revoke-keys"
Oct 30 21:29:37.290973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.288931    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62e800f-d9a7-4539-84f6-ff97f5229123" containerName="revoke-keys"
Oct 30 21:29:37.291881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.291842    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62e800f-d9a7-4539-84f6-ff97f5229123" containerName="revoke-keys"
Oct 30 21:29:37.312169 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.312138    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.330875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.317377    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:29:37.330875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.317464    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:37.330875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.323276    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"]
Oct 30 21:29:37.410992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.410951    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnxdr\" (UniqueName: \"kubernetes.io/projected/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-kube-api-access-fnxdr\") pod \"maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.411186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.411121    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.411186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.411167    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.515468 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.515422    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnxdr\" (UniqueName: \"kubernetes.io/projected/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-kube-api-access-fnxdr\") pod \"maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.515691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.515536    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.515691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.515567    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.516388 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.516352    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.518191 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.518167    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.524585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.524561    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnxdr\" (UniqueName: \"kubernetes.io/projected/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-kube-api-access-fnxdr\") pod \"maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.661837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.661805    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:37.950743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:37.950705    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"]
Oct 30 21:29:37.951573 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:29:37.951543    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8062a4d5_5fc5_4c4e_949d_f74659ff6ba9.slice/crio-7ffd389f31732a9133c639f0ecbe0c95dcc598ae8d5e466e552e9fdfb3c74752 WatchSource:0}: Error finding container 7ffd389f31732a9133c639f0ecbe0c95dcc598ae8d5e466e552e9fdfb3c74752: Status 404 returned error can't find the container with id 7ffd389f31732a9133c639f0ecbe0c95dcc598ae8d5e466e552e9fdfb3c74752
Oct 30 21:29:38.859805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:38.859719    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq" event={"ID":"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9","Type":"ContainerStarted","Data":"8ab41d3ce6b12e203222e2a65e74bc3b86608ce8cc8132a5e0f30a89f7862ad2"}
Oct 30 21:29:38.859805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:38.859792    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq" event={"ID":"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9","Type":"ContainerStarted","Data":"7ffd389f31732a9133c639f0ecbe0c95dcc598ae8d5e466e552e9fdfb3c74752"}
Oct 30 21:29:38.902879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:38.882126    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq" podStartSLOduration=1.8821082169999999 podStartE2EDuration="1.882108217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:38.876903413 +0000 UTC m=+3145.368719958" watchObservedRunningTime="2026-10-30 21:29:38.882108217 +0000 UTC m=+3145.373924759"
Oct 30 21:29:39.869496 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:39.869461    4810 generic.go:356] "Generic (PLEG): container finished" podID="8062a4d5-5fc5-4c4e-949d-f74659ff6ba9" containerID="8ab41d3ce6b12e203222e2a65e74bc3b86608ce8cc8132a5e0f30a89f7862ad2" exitCode=0
Oct 30 21:29:39.869921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:39.869543    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq" event={"ID":"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9","Type":"ContainerDied","Data":"8ab41d3ce6b12e203222e2a65e74bc3b86608ce8cc8132a5e0f30a89f7862ad2"}
Oct 30 21:29:40.949389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:40.949354    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t" podUID="3fb4beb5-9cef-4b11-add6-c18470c22c09" containerName="main" containerID="cri-o://67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c" gracePeriod=45
Oct 30 21:29:41.095258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.094963    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:41.217318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.213115    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-xks-ca\") pod \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") "
Oct 30 21:29:41.217318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.213186    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-openshift-ca\") pod \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") "
Oct 30 21:29:41.217318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.213351    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-kube-api-access-fnxdr\" (UniqueName: \"kubernetes.io/projected/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-kube-api-access-fnxdr\") pod \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\" (UID: \"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9\") "
Oct 30 21:29:41.217318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.214613    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-openshift-ca" pod "8062a4d5-5fc5-4c4e-949d-f74659ff6ba9" (UID: "8062a4d5-5fc5-4c4e-949d-f74659ff6ba9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:41.219032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.219001    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-xks-ca" pod "8062a4d5-5fc5-4c4e-949d-f74659ff6ba9" (UID: "8062a4d5-5fc5-4c4e-949d-f74659ff6ba9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:41.226051 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.226022    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-kube-api-access-fnxdr" pod "8062a4d5-5fc5-4c4e-949d-f74659ff6ba9" (UID: "8062a4d5-5fc5-4c4e-949d-f74659ff6ba9"). InnerVolumeSpecName "kube-api-access-fnxdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:41.253497 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.253475    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:29:41.315280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.315240    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-home\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-home\") pod \"3fb4beb5-9cef-4b11-add6-c18470c22c09\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") "
Oct 30 21:29:41.315715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.315337    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-tmp-dir\") pod \"3fb4beb5-9cef-4b11-add6-c18470c22c09\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") "
Oct 30 21:29:41.315715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.315405    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb4beb5-9cef-4b11-add6-c18470c22c09-kube-api-access-7xk86\" (UniqueName: \"kubernetes.io/projected/3fb4beb5-9cef-4b11-add6-c18470c22c09-kube-api-access-7xk86\") pod \"3fb4beb5-9cef-4b11-add6-c18470c22c09\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") "
Oct 30 21:29:41.315715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.315436    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-dshm\") pod \"3fb4beb5-9cef-4b11-add6-c18470c22c09\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") "
Oct 30 21:29:41.315715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.315471    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-model-cache\") pod \"3fb4beb5-9cef-4b11-add6-c18470c22c09\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") "
Oct 30 21:29:41.315715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.315589    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb4beb5-9cef-4b11-add6-c18470c22c09-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb4beb5-9cef-4b11-add6-c18470c22c09-tls-certs\") pod \"3fb4beb5-9cef-4b11-add6-c18470c22c09\" (UID: \"3fb4beb5-9cef-4b11-add6-c18470c22c09\") "
Oct 30 21:29:41.316322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.316295    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnxdr\" (UniqueName: \"kubernetes.io/projected/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-kube-api-access-fnxdr\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:41.316446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.316326    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:41.316446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.316340    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:41.317273 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.317241    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-home" pod "3fb4beb5-9cef-4b11-add6-c18470c22c09" (UID: "3fb4beb5-9cef-4b11-add6-c18470c22c09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:41.317374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.317351    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-model-cache" pod "3fb4beb5-9cef-4b11-add6-c18470c22c09" (UID: "3fb4beb5-9cef-4b11-add6-c18470c22c09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:41.317447 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.317367    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-tmp-dir" pod "3fb4beb5-9cef-4b11-add6-c18470c22c09" (UID: "3fb4beb5-9cef-4b11-add6-c18470c22c09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:41.320248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.320217    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb4beb5-9cef-4b11-add6-c18470c22c09-kube-api-access-7xk86" pod "3fb4beb5-9cef-4b11-add6-c18470c22c09" (UID: "3fb4beb5-9cef-4b11-add6-c18470c22c09"). InnerVolumeSpecName "kube-api-access-7xk86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:41.320596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.320569    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-dshm" pod "3fb4beb5-9cef-4b11-add6-c18470c22c09" (UID: "3fb4beb5-9cef-4b11-add6-c18470c22c09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:41.322353 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.322326    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb4beb5-9cef-4b11-add6-c18470c22c09-tls-certs" pod "3fb4beb5-9cef-4b11-add6-c18470c22c09" (UID: "3fb4beb5-9cef-4b11-add6-c18470c22c09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:41.418079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.417987    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:41.418079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.418021    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xk86\" (UniqueName: \"kubernetes.io/projected/3fb4beb5-9cef-4b11-add6-c18470c22c09-kube-api-access-7xk86\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:41.418079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.418035    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:41.418079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.418048    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:41.418079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.418060    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb4beb5-9cef-4b11-add6-c18470c22c09-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:41.418079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.418072    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb4beb5-9cef-4b11-add6-c18470c22c09-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:41.894375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.894346    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"
Oct 30 21:29:41.894560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.894384    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq" event={"ID":"8062a4d5-5fc5-4c4e-949d-f74659ff6ba9","Type":"ContainerDied","Data":"7ffd389f31732a9133c639f0ecbe0c95dcc598ae8d5e466e552e9fdfb3c74752"}
Oct 30 21:29:41.894560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.894432    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ffd389f31732a9133c639f0ecbe0c95dcc598ae8d5e466e552e9fdfb3c74752"
Oct 30 21:29:41.896266 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.896242    4810 generic.go:356] "Generic (PLEG): container finished" podID="3fb4beb5-9cef-4b11-add6-c18470c22c09" containerID="67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c" exitCode=0
Oct 30 21:29:41.896402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.896284    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t" event={"ID":"3fb4beb5-9cef-4b11-add6-c18470c22c09","Type":"ContainerDied","Data":"67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c"}
Oct 30 21:29:41.896402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.896310    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t" event={"ID":"3fb4beb5-9cef-4b11-add6-c18470c22c09","Type":"ContainerDied","Data":"4338eba1af8bc59ecf97cffa197d45a3eb979cef7f7929d4adbdeb810f62f0a5"}
Oct 30 21:29:41.896402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.896313    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"
Oct 30 21:29:41.896402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.896331    4810 scope.go:122] "RemoveContainer" containerID="67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c"
Oct 30 21:29:41.911580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.911555    4810 scope.go:122] "RemoveContainer" containerID="67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c"
Oct 30 21:29:41.911940 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:29:41.911911    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c\": container with ID starting with 67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c not found: ID does not exist" containerID="67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c"
Oct 30 21:29:41.912051 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.911950    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c"} err="failed to get container status \"67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c\": rpc error: code = NotFound desc = could not find container \"67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c\": container with ID starting with 67765bbcc716c55c1bd57028998b40ff9703691da707d816939aa907c7670b2c not found: ID does not exist"
Oct 30 21:29:41.923802 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.923779    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"]
Oct 30 21:29:41.929621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:41.929599    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ffce13/test-model-ffce13-kserve-5d595c6db7-j624t"]
Oct 30 21:29:42.365805 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:42.365718    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"]
Oct 30 21:29:42.366291 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:42.365934    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerName="istio-proxy" containerID="cri-o://115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06" gracePeriod=30
Oct 30 21:29:43.666834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:43.666797    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb4beb5-9cef-4b11-add6-c18470c22c09" path="/var/lib/kubelet/pods/3fb4beb5-9cef-4b11-add6-c18470c22c09/volumes"
Oct 30 21:29:44.727490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.727458    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"]
Oct 30 21:29:44.728253 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.728237    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3fb4beb5-9cef-4b11-add6-c18470c22c09" containerName="main"
Oct 30 21:29:44.728314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.728254    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb4beb5-9cef-4b11-add6-c18470c22c09" containerName="main"
Oct 30 21:29:44.728353 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.728316    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8062a4d5-5fc5-4c4e-949d-f74659ff6ba9" containerName="revoke-keys"
Oct 30 21:29:44.728353 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.728321    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8062a4d5-5fc5-4c4e-949d-f74659ff6ba9" containerName="revoke-keys"
Oct 30 21:29:44.728446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.728437    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8062a4d5-5fc5-4c4e-949d-f74659ff6ba9" containerName="revoke-keys"
Oct 30 21:29:44.728483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.728450    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb4beb5-9cef-4b11-add6-c18470c22c09" containerName="main"
Oct 30 21:29:44.732905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.732319    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:44.746936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.737123    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:44.746936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.737205    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:29:44.746936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.742961    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"]
Oct 30 21:29:44.868417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.868379    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgn8z\" (UniqueName: \"kubernetes.io/projected/f1115536-edb7-4c90-a90a-6015b87dff41-kube-api-access-vgn8z\") pod \"maas-api-revoke-sub-fde3b9a6-gqghn\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:44.868604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.868490    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde3b9a6-gqghn\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:44.868604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.868602    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde3b9a6-gqghn\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:44.970644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.970606    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde3b9a6-gqghn\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:44.970827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.970751    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde3b9a6-gqghn\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:44.970994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.970978    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgn8z\" (UniqueName: \"kubernetes.io/projected/f1115536-edb7-4c90-a90a-6015b87dff41-kube-api-access-vgn8z\") pod \"maas-api-revoke-sub-fde3b9a6-gqghn\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:44.971479 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.971452    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde3b9a6-gqghn\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:44.973329 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.973303    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde3b9a6-gqghn\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:44.979849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:44.979792    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgn8z\" (UniqueName: \"kubernetes.io/projected/f1115536-edb7-4c90-a90a-6015b87dff41-kube-api-access-vgn8z\") pod \"maas-api-revoke-sub-fde3b9a6-gqghn\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:45.076756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:45.076728    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:45.362678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:45.362647    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"]
Oct 30 21:29:45.363040 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:29:45.363014    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1115536_edb7_4c90_a90a_6015b87dff41.slice/crio-10c2419d83bf386676844889f5a306dec16898caccd1ad6eb5a5891f37d92e35 WatchSource:0}: Error finding container 10c2419d83bf386676844889f5a306dec16898caccd1ad6eb5a5891f37d92e35: Status 404 returned error can't find the container with id 10c2419d83bf386676844889f5a306dec16898caccd1ad6eb5a5891f37d92e35
Oct 30 21:29:45.950689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:45.950600    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn" event={"ID":"f1115536-edb7-4c90-a90a-6015b87dff41","Type":"ContainerStarted","Data":"07a0eecb43de0e30650cd94cb5d5a30a020286e15fb35505a9a5d244ca91f10c"}
Oct 30 21:29:45.950689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:45.950649    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn" event={"ID":"f1115536-edb7-4c90-a90a-6015b87dff41","Type":"ContainerStarted","Data":"10c2419d83bf386676844889f5a306dec16898caccd1ad6eb5a5891f37d92e35"}
Oct 30 21:29:45.970785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:45.970730    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn" podStartSLOduration=1.9707132010000001 podStartE2EDuration="1.970713201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:45.967723895 +0000 UTC m=+3152.459540441" watchObservedRunningTime="2026-10-30 21:29:45.970713201 +0000 UTC m=+3152.462529747"
Oct 30 21:29:46.962163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:46.962130    4810 generic.go:356] "Generic (PLEG): container finished" podID="f1115536-edb7-4c90-a90a-6015b87dff41" containerID="07a0eecb43de0e30650cd94cb5d5a30a020286e15fb35505a9a5d244ca91f10c" exitCode=0
Oct 30 21:29:46.962624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:46.962219    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn" event={"ID":"f1115536-edb7-4c90-a90a-6015b87dff41","Type":"ContainerDied","Data":"07a0eecb43de0e30650cd94cb5d5a30a020286e15fb35505a9a5d244ca91f10c"}
Oct 30 21:29:47.433871 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.433827    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"]
Oct 30 21:29:47.445808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.445779    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"]
Oct 30 21:29:47.730670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.730642    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:29:47.847884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.833231    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-envoy\") pod \"84725bf3-ddbc-49b1-91aa-602ab24f206f\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") "
Oct 30 21:29:47.847884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.840891    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84725bf3-ddbc-49b1-91aa-602ab24f206f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84725bf3-ddbc-49b1-91aa-602ab24f206f-istiod-ca-cert\") pod \"84725bf3-ddbc-49b1-91aa-602ab24f206f\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") "
Oct 30 21:29:47.847884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.841245    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-credential-socket" pod "84725bf3-ddbc-49b1-91aa-602ab24f206f" (UID: "84725bf3-ddbc-49b1-91aa-602ab24f206f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:47.871881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.862706    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84725bf3-ddbc-49b1-91aa-602ab24f206f-istiod-ca-cert" pod "84725bf3-ddbc-49b1-91aa-602ab24f206f" (UID: "84725bf3-ddbc-49b1-91aa-602ab24f206f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:47.871881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.862781    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-credential-socket\") pod \"84725bf3-ddbc-49b1-91aa-602ab24f206f\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") "
Oct 30 21:29:47.871881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.862907    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-token\" (UniqueName: \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-token\") pod \"84725bf3-ddbc-49b1-91aa-602ab24f206f\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") "
Oct 30 21:29:47.894881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.863002    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-kube-api-access-bk64b\" (UniqueName: \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-kube-api-access-bk64b\") pod \"84725bf3-ddbc-49b1-91aa-602ab24f206f\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") "
Oct 30 21:29:47.894881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.894404    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-podinfo\") pod \"84725bf3-ddbc-49b1-91aa-602ab24f206f\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") "
Oct 30 21:29:47.894881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.894519    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-socket\") pod \"84725bf3-ddbc-49b1-91aa-602ab24f206f\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") "
Oct 30 21:29:47.894881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.894598    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-data\") pod \"84725bf3-ddbc-49b1-91aa-602ab24f206f\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") "
Oct 30 21:29:47.894881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.894629    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-certs\") pod \"84725bf3-ddbc-49b1-91aa-602ab24f206f\" (UID: \"84725bf3-ddbc-49b1-91aa-602ab24f206f\") "
Oct 30 21:29:47.905924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.895426    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84725bf3-ddbc-49b1-91aa-602ab24f206f-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:47.905924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.895451    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:47.905924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.896070    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-data" pod "84725bf3-ddbc-49b1-91aa-602ab24f206f" (UID: "84725bf3-ddbc-49b1-91aa-602ab24f206f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:47.905924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.896364    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-socket" pod "84725bf3-ddbc-49b1-91aa-602ab24f206f" (UID: "84725bf3-ddbc-49b1-91aa-602ab24f206f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:47.905924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.896426    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-certs" pod "84725bf3-ddbc-49b1-91aa-602ab24f206f" (UID: "84725bf3-ddbc-49b1-91aa-602ab24f206f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:47.905924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.901637    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-token" pod "84725bf3-ddbc-49b1-91aa-602ab24f206f" (UID: "84725bf3-ddbc-49b1-91aa-602ab24f206f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:47.912676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.912599    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-podinfo" pod "84725bf3-ddbc-49b1-91aa-602ab24f206f" (UID: "84725bf3-ddbc-49b1-91aa-602ab24f206f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:29:47.916553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.916515    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-kube-api-access-bk64b" pod "84725bf3-ddbc-49b1-91aa-602ab24f206f" (UID: "84725bf3-ddbc-49b1-91aa-602ab24f206f"). InnerVolumeSpecName "kube-api-access-bk64b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:47.916984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.916964    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-envoy" pod "84725bf3-ddbc-49b1-91aa-602ab24f206f" (UID: "84725bf3-ddbc-49b1-91aa-602ab24f206f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:47.973027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.972947    4810 generic.go:356] "Generic (PLEG): container finished" podID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerID="115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06" exitCode=0
Oct 30 21:29:47.973027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.973000    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"
Oct 30 21:29:47.973471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.973037    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" event={"ID":"84725bf3-ddbc-49b1-91aa-602ab24f206f","Type":"ContainerDied","Data":"115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06"}
Oct 30 21:29:47.973471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.973103    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp" event={"ID":"84725bf3-ddbc-49b1-91aa-602ab24f206f","Type":"ContainerDied","Data":"9869151ddf7e85d92d202537d00a000ac7861fa5269ba9d95e4ce9c8c7af4f48"}
Oct 30 21:29:47.973471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.973131    4810 scope.go:122] "RemoveContainer" containerID="115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06"
Oct 30 21:29:47.988337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.988316    4810 scope.go:122] "RemoveContainer" containerID="115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06"
Oct 30 21:29:47.988646 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:29:47.988624    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06\": container with ID starting with 115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06 not found: ID does not exist" containerID="115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06"
Oct 30 21:29:47.988723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.988655    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06"} err="failed to get container status \"115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06\": rpc error: code = NotFound desc = could not find container \"115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06\": container with ID starting with 115380c28c9f2aba240b637c24fa5492ebf056f9897a337055cccb7525708c06 not found: ID does not exist"
Oct 30 21:29:47.997543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.997518    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:47.997643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.997546    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:47.997643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.997557    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:47.997643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.997565    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:47.997643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.997574    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:47.997643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.997583    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk64b\" (UniqueName: \"kubernetes.io/projected/84725bf3-ddbc-49b1-91aa-602ab24f206f-kube-api-access-bk64b\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:47.997643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:47.997593    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84725bf3-ddbc-49b1-91aa-602ab24f206f-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:48.004736 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.004712    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"]
Oct 30 21:29:48.010729 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.010703    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-69ckp"]
Oct 30 21:29:48.148128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.148102    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:48.199771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.199731    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-openshift-ca\") pod \"f1115536-edb7-4c90-a90a-6015b87dff41\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") "
Oct 30 21:29:48.199998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.199805    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-xks-ca\") pod \"f1115536-edb7-4c90-a90a-6015b87dff41\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") "
Oct 30 21:29:48.199998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.199883    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1115536-edb7-4c90-a90a-6015b87dff41-kube-api-access-vgn8z\" (UniqueName: \"kubernetes.io/projected/f1115536-edb7-4c90-a90a-6015b87dff41-kube-api-access-vgn8z\") pod \"f1115536-edb7-4c90-a90a-6015b87dff41\" (UID: \"f1115536-edb7-4c90-a90a-6015b87dff41\") "
Oct 30 21:29:48.200182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.200155    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-openshift-ca" pod "f1115536-edb7-4c90-a90a-6015b87dff41" (UID: "f1115536-edb7-4c90-a90a-6015b87dff41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:48.200680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.200658    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:48.202840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.202810    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-xks-ca" pod "f1115536-edb7-4c90-a90a-6015b87dff41" (UID: "f1115536-edb7-4c90-a90a-6015b87dff41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:48.202840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.202829    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1115536-edb7-4c90-a90a-6015b87dff41-kube-api-access-vgn8z" pod "f1115536-edb7-4c90-a90a-6015b87dff41" (UID: "f1115536-edb7-4c90-a90a-6015b87dff41"). InnerVolumeSpecName "kube-api-access-vgn8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:48.301809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.301726    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1115536-edb7-4c90-a90a-6015b87dff41-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:48.301809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.301760    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgn8z\" (UniqueName: \"kubernetes.io/projected/f1115536-edb7-4c90-a90a-6015b87dff41-kube-api-access-vgn8z\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:48.984163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.984131    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"
Oct 30 21:29:48.984551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.984132    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn" event={"ID":"f1115536-edb7-4c90-a90a-6015b87dff41","Type":"ContainerDied","Data":"10c2419d83bf386676844889f5a306dec16898caccd1ad6eb5a5891f37d92e35"}
Oct 30 21:29:48.984551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:48.984287    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10c2419d83bf386676844889f5a306dec16898caccd1ad6eb5a5891f37d92e35"
Oct 30 21:29:49.664566 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:49.664529    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" path="/var/lib/kubelet/pods/84725bf3-ddbc-49b1-91aa-602ab24f206f/volumes"
Oct 30 21:29:52.054491 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.054450    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq" podUID="130112da-8af4-44a1-99bf-4728f09414d0" containerName="main" containerID="cri-o://664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270" gracePeriod=45
Oct 30 21:29:52.363345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.363308    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:29:52.435208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.435163    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b" podUID="33b2eef0-0705-463e-9c89-96f86d0c1576" containerName="payload-processing" containerID="cri-o://9d0d3b76b844914770f7f46c2d70f855f9595098e2fb5ede719791adb78b802d" gracePeriod=25
Oct 30 21:29:52.446720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.446684    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9" podUID="b92d5aab-dbcf-47e0-a2c9-111af9c207c6" containerName="payload-pre-processing" containerID="cri-o://743d59e793a1835ec8fe8945d8604c54269fb7db392888046a60c9190a182a46" gracePeriod=25
Oct 30 21:29:52.466702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.466668    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-model-cache\") pod \"130112da-8af4-44a1-99bf-4728f09414d0\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") "
Oct 30 21:29:52.466836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.466713    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130112da-8af4-44a1-99bf-4728f09414d0-kube-api-access-xnx5m\" (UniqueName: \"kubernetes.io/projected/130112da-8af4-44a1-99bf-4728f09414d0-kube-api-access-xnx5m\") pod \"130112da-8af4-44a1-99bf-4728f09414d0\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") "
Oct 30 21:29:52.466949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.466899    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/130112da-8af4-44a1-99bf-4728f09414d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/130112da-8af4-44a1-99bf-4728f09414d0-tls-certs\") pod \"130112da-8af4-44a1-99bf-4728f09414d0\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") "
Oct 30 21:29:52.467015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.466960    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-model-cache" pod "130112da-8af4-44a1-99bf-4728f09414d0" (UID: "130112da-8af4-44a1-99bf-4728f09414d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:52.467015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.466992    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-tmp-dir\") pod \"130112da-8af4-44a1-99bf-4728f09414d0\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") "
Oct 30 21:29:52.467149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.467058    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-dshm\") pod \"130112da-8af4-44a1-99bf-4728f09414d0\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") "
Oct 30 21:29:52.467149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.467103    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-home\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-home\") pod \"130112da-8af4-44a1-99bf-4728f09414d0\" (UID: \"130112da-8af4-44a1-99bf-4728f09414d0\") "
Oct 30 21:29:52.467262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.467195    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-tmp-dir" pod "130112da-8af4-44a1-99bf-4728f09414d0" (UID: "130112da-8af4-44a1-99bf-4728f09414d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:52.467481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.467459    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-home" pod "130112da-8af4-44a1-99bf-4728f09414d0" (UID: "130112da-8af4-44a1-99bf-4728f09414d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:52.467715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.467700    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:52.467784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.467717    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:52.467784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.467726    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:52.469132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.469109    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130112da-8af4-44a1-99bf-4728f09414d0-tls-certs" pod "130112da-8af4-44a1-99bf-4728f09414d0" (UID: "130112da-8af4-44a1-99bf-4728f09414d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:52.469326 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.469312    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-dshm" pod "130112da-8af4-44a1-99bf-4728f09414d0" (UID: "130112da-8af4-44a1-99bf-4728f09414d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:52.469410 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.469387    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130112da-8af4-44a1-99bf-4728f09414d0-kube-api-access-xnx5m" pod "130112da-8af4-44a1-99bf-4728f09414d0" (UID: "130112da-8af4-44a1-99bf-4728f09414d0"). InnerVolumeSpecName "kube-api-access-xnx5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:52.569411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.569374    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130112da-8af4-44a1-99bf-4728f09414d0-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:52.569411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.569408    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130112da-8af4-44a1-99bf-4728f09414d0-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:52.569411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:52.569419    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnx5m\" (UniqueName: \"kubernetes.io/projected/130112da-8af4-44a1-99bf-4728f09414d0-kube-api-access-xnx5m\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:53.022792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:53.022758    4810 generic.go:356] "Generic (PLEG): container finished" podID="130112da-8af4-44a1-99bf-4728f09414d0" containerID="664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270" exitCode=0
Oct 30 21:29:53.023005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:53.022801    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq" event={"ID":"130112da-8af4-44a1-99bf-4728f09414d0","Type":"ContainerDied","Data":"664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270"}
Oct 30 21:29:53.023005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:53.022828    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq" event={"ID":"130112da-8af4-44a1-99bf-4728f09414d0","Type":"ContainerDied","Data":"1dd8decbfe5e14327240c973675ec44a9f317ddf632f8af5ea3eae82dafcdfae"}
Oct 30 21:29:53.023005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:53.022845    4810 scope.go:122] "RemoveContainer" containerID="664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270"
Oct 30 21:29:53.023005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:53.022802    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"
Oct 30 21:29:53.034743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:53.034726    4810 scope.go:122] "RemoveContainer" containerID="664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270"
Oct 30 21:29:53.035021 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:29:53.034997    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270\": container with ID starting with 664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270 not found: ID does not exist" containerID="664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270"
Oct 30 21:29:53.035088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:53.035027    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270"} err="failed to get container status \"664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270\": rpc error: code = NotFound desc = could not find container \"664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270\": container with ID starting with 664639e8d46c12bd94ac3adae36ca991d7a261a03cc2948d97731f8f22449270 not found: ID does not exist"
Oct 30 21:29:53.048013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:53.047984    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"]
Oct 30 21:29:53.051937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:53.051914    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9b78e4/test-model-9b78e4-kserve-99676f64f-xx7lq"]
Oct 30 21:29:53.664833 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:53.664795    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130112da-8af4-44a1-99bf-4728f09414d0" path="/var/lib/kubelet/pods/130112da-8af4-44a1-99bf-4728f09414d0/volumes"
Oct 30 21:29:55.157803 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:29:55.157771    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:29:55.564811 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.564719    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"]
Oct 30 21:29:55.567615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.566673    4810 cpu_manager.go:414] "containerMap: removing container" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerName="istio-proxy"
Oct 30 21:29:55.567615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.566701    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerName="istio-proxy"
Oct 30 21:29:55.567615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.566736    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f1115536-edb7-4c90-a90a-6015b87dff41" containerName="revoke-keys"
Oct 30 21:29:55.567615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.566745    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1115536-edb7-4c90-a90a-6015b87dff41" containerName="revoke-keys"
Oct 30 21:29:55.567615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.566804    4810 cpu_manager.go:414] "containerMap: removing container" podUID="130112da-8af4-44a1-99bf-4728f09414d0" containerName="main"
Oct 30 21:29:55.567615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.566813    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130112da-8af4-44a1-99bf-4728f09414d0" containerName="main"
Oct 30 21:29:55.567615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.567070    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="130112da-8af4-44a1-99bf-4728f09414d0" containerName="main"
Oct 30 21:29:55.567615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.567099    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="84725bf3-ddbc-49b1-91aa-602ab24f206f" containerName="istio-proxy"
Oct 30 21:29:55.567615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.567113    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1115536-edb7-4c90-a90a-6015b87dff41" containerName="revoke-keys"
Oct 30 21:29:55.573178 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.573156    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:55.580495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.580248    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:55.580495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.580341    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:29:55.600020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.599986    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"]
Oct 30 21:29:55.647876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.647420    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ac526-plskw\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:55.647876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.647574    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsscb\" (UniqueName: \"kubernetes.io/projected/eb0a0666-bb04-4253-b9a9-bf15716f3db6-kube-api-access-xsscb\") pod \"maas-api-revoke-keys-e2e-inf-a-2ac526-plskw\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:55.647876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.647729    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ac526-plskw\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:55.748812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.748769    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ac526-plskw\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:55.749078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.748841    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ac526-plskw\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:55.749318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.749289    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsscb\" (UniqueName: \"kubernetes.io/projected/eb0a0666-bb04-4253-b9a9-bf15716f3db6-kube-api-access-xsscb\") pod \"maas-api-revoke-keys-e2e-inf-a-2ac526-plskw\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:55.749589 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.749566    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ac526-plskw\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:55.752741 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.752539    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2ac526-plskw\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:55.757786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.757760    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsscb\" (UniqueName: \"kubernetes.io/projected/eb0a0666-bb04-4253-b9a9-bf15716f3db6-kube-api-access-xsscb\") pod \"maas-api-revoke-keys-e2e-inf-a-2ac526-plskw\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:55.928537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:55.928503    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:56.225124 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:29:56.225089    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0a0666_bb04_4253_b9a9_bf15716f3db6.slice/crio-ad34e66a2c2e7e2f7da6ec09da4e11a432424d3966dd51d4df15b889afaa518c WatchSource:0}: Error finding container ad34e66a2c2e7e2f7da6ec09da4e11a432424d3966dd51d4df15b889afaa518c: Status 404 returned error can't find the container with id ad34e66a2c2e7e2f7da6ec09da4e11a432424d3966dd51d4df15b889afaa518c
Oct 30 21:29:56.225652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:56.225628    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"]
Oct 30 21:29:57.075801 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:57.075763    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw" event={"ID":"eb0a0666-bb04-4253-b9a9-bf15716f3db6","Type":"ContainerStarted","Data":"fa0be4dca6ba2051edb0d6b687b6633b6f215fb4e2a186c009e96987ea28ce27"}
Oct 30 21:29:57.076052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:57.075824    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw" event={"ID":"eb0a0666-bb04-4253-b9a9-bf15716f3db6","Type":"ContainerStarted","Data":"ad34e66a2c2e7e2f7da6ec09da4e11a432424d3966dd51d4df15b889afaa518c"}
Oct 30 21:29:57.094888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:57.094814    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw" podStartSLOduration=2.094801293 podStartE2EDuration="2.094801293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:57.090566662 +0000 UTC m=+3163.582383206" watchObservedRunningTime="2026-10-30 21:29:57.094801293 +0000 UTC m=+3163.586617838"
Oct 30 21:29:58.088088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:58.088052    4810 generic.go:356] "Generic (PLEG): container finished" podID="eb0a0666-bb04-4253-b9a9-bf15716f3db6" containerID="fa0be4dca6ba2051edb0d6b687b6633b6f215fb4e2a186c009e96987ea28ce27" exitCode=0
Oct 30 21:29:58.088505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:58.088144    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw" event={"ID":"eb0a0666-bb04-4253-b9a9-bf15716f3db6","Type":"ContainerDied","Data":"fa0be4dca6ba2051edb0d6b687b6633b6f215fb4e2a186c009e96987ea28ce27"}
Oct 30 21:29:59.281817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:59.281786    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:29:59.315851 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:59.315822    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-openshift-ca\") pod \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") "
Oct 30 21:29:59.316063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:59.315999    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0a0666-bb04-4253-b9a9-bf15716f3db6-kube-api-access-xsscb\" (UniqueName: \"kubernetes.io/projected/eb0a0666-bb04-4253-b9a9-bf15716f3db6-kube-api-access-xsscb\") pod \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") "
Oct 30 21:29:59.316133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:59.316102    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-xks-ca\") pod \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\" (UID: \"eb0a0666-bb04-4253-b9a9-bf15716f3db6\") "
Oct 30 21:29:59.316246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:59.316218    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-openshift-ca" pod "eb0a0666-bb04-4253-b9a9-bf15716f3db6" (UID: "eb0a0666-bb04-4253-b9a9-bf15716f3db6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:59.316709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:59.316680    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:59.318614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:59.318586    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0a0666-bb04-4253-b9a9-bf15716f3db6-kube-api-access-xsscb" pod "eb0a0666-bb04-4253-b9a9-bf15716f3db6" (UID: "eb0a0666-bb04-4253-b9a9-bf15716f3db6"). InnerVolumeSpecName "kube-api-access-xsscb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:59.320007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:59.319980    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-xks-ca" pod "eb0a0666-bb04-4253-b9a9-bf15716f3db6" (UID: "eb0a0666-bb04-4253-b9a9-bf15716f3db6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:59.417579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:59.417550    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0a0666-bb04-4253-b9a9-bf15716f3db6-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:59.417579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:29:59.417578    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsscb\" (UniqueName: \"kubernetes.io/projected/eb0a0666-bb04-4253-b9a9-bf15716f3db6-kube-api-access-xsscb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:00.117725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.117637    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"
Oct 30 21:30:00.117725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.117657    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw" event={"ID":"eb0a0666-bb04-4253-b9a9-bf15716f3db6","Type":"ContainerDied","Data":"ad34e66a2c2e7e2f7da6ec09da4e11a432424d3966dd51d4df15b889afaa518c"}
Oct 30 21:30:00.117725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.117710    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad34e66a2c2e7e2f7da6ec09da4e11a432424d3966dd51d4df15b889afaa518c"
Oct 30 21:30:00.152588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.148157    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"]
Oct 30 21:30:00.152588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.149390    4810 cpu_manager.go:414] "containerMap: removing container" podUID="eb0a0666-bb04-4253-b9a9-bf15716f3db6" containerName="revoke-keys"
Oct 30 21:30:00.152588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.149431    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0a0666-bb04-4253-b9a9-bf15716f3db6" containerName="revoke-keys"
Oct 30 21:30:00.152588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.149683    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0a0666-bb04-4253-b9a9-bf15716f3db6" containerName="revoke-keys"
Oct 30 21:30:00.156879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.156347    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"
Oct 30 21:30:00.203908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.201068    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:30:00.222942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.222907    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"]
Oct 30 21:30:00.232669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.232312    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q66qt\" (UniqueName: \"kubernetes.io/projected/7a4e9f94-c578-43a7-9f01-937d5cc8c679-kube-api-access-q66qt\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css\" (UID: \"7a4e9f94-c578-43a7-9f01-937d5cc8c679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"
Oct 30 21:30:00.232669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.232366    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a4e9f94-c578-43a7-9f01-937d5cc8c679-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css\" (UID: \"7a4e9f94-c578-43a7-9f01-937d5cc8c679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"
Oct 30 21:30:00.232669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.232408    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"
Oct 30 21:30:00.233791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.232328    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"]
Oct 30 21:30:00.246328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.246290    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"]
Oct 30 21:30:00.246604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.246422    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"
Oct 30 21:30:00.294131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.293981    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"]
Oct 30 21:30:00.294131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.294096    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"
Oct 30 21:30:00.298694 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.298650    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"]
Oct 30 21:30:00.318375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.316438    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"]
Oct 30 21:30:00.334905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.334626    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/500bb411-3bf5-4e1a-bc15-36d332b0af13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl\" (UID: \"500bb411-3bf5-4e1a-bc15-36d332b0af13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"
Oct 30 21:30:00.334905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.334676    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp27b\" (UniqueName: \"kubernetes.io/projected/500bb411-3bf5-4e1a-bc15-36d332b0af13-kube-api-access-qp27b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl\" (UID: \"500bb411-3bf5-4e1a-bc15-36d332b0af13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"
Oct 30 21:30:00.334905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.334764    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnjg\" (UniqueName: \"kubernetes.io/projected/8fd73835-f5e2-47b8-8403-3aa7f03492eb-kube-api-access-jvnjg\") pod \"maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7\" (UID: \"8fd73835-f5e2-47b8-8403-3aa7f03492eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"
Oct 30 21:30:00.334905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.334817    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf881a3-86a2-494d-b8e5-3a789801bd41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp\" (UID: \"aaf881a3-86a2-494d-b8e5-3a789801bd41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"
Oct 30 21:30:00.334905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.334892    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qf98\" (UniqueName: \"kubernetes.io/projected/aaf881a3-86a2-494d-b8e5-3a789801bd41-kube-api-access-4qf98\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp\" (UID: \"aaf881a3-86a2-494d-b8e5-3a789801bd41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"
Oct 30 21:30:00.335321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.334951    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q66qt\" (UniqueName: \"kubernetes.io/projected/7a4e9f94-c578-43a7-9f01-937d5cc8c679-kube-api-access-q66qt\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css\" (UID: \"7a4e9f94-c578-43a7-9f01-937d5cc8c679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"
Oct 30 21:30:00.335321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.334994    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a4e9f94-c578-43a7-9f01-937d5cc8c679-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css\" (UID: \"7a4e9f94-c578-43a7-9f01-937d5cc8c679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"
Oct 30 21:30:00.335321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.335159    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd73835-f5e2-47b8-8403-3aa7f03492eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7\" (UID: \"8fd73835-f5e2-47b8-8403-3aa7f03492eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"
Oct 30 21:30:00.336470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.336345    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a4e9f94-c578-43a7-9f01-937d5cc8c679-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css\" (UID: \"7a4e9f94-c578-43a7-9f01-937d5cc8c679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"
Oct 30 21:30:00.339884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.339745    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"]
Oct 30 21:30:00.343455 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.339844    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"
Oct 30 21:30:00.350197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.350010    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"]
Oct 30 21:30:00.350197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.350060    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"]
Oct 30 21:30:00.351061 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.350313    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"
Oct 30 21:30:00.359490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.359463    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"]
Oct 30 21:30:00.359608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.359571    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"
Oct 30 21:30:00.362063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.362042    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"]
Oct 30 21:30:00.363268 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.363243    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q66qt\" (UniqueName: \"kubernetes.io/projected/7a4e9f94-c578-43a7-9f01-937d5cc8c679-kube-api-access-q66qt\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css\" (UID: \"7a4e9f94-c578-43a7-9f01-937d5cc8c679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"
Oct 30 21:30:00.366470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.366448    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"]
Oct 30 21:30:00.379055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.379035    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"]
Oct 30 21:30:00.379228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.379104    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"
Oct 30 21:30:00.382349 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.382318    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"]
Oct 30 21:30:00.393561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.393535    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"]
Oct 30 21:30:00.399098 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.399077    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"
Oct 30 21:30:00.400934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.400911    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"]
Oct 30 21:30:00.405311 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.405288    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"]
Oct 30 21:30:00.420063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.420038    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"]
Oct 30 21:30:00.420190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.420125    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"
Oct 30 21:30:00.426106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.426079    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"]
Oct 30 21:30:00.441413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.437519    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj6fn\" (UniqueName: \"kubernetes.io/projected/49e54b96-7b3a-421c-9ca0-312047e92c64-kube-api-access-zj6fn\") pod \"maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62\" (UID: \"49e54b96-7b3a-421c-9ca0-312047e92c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"
Oct 30 21:30:00.441413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.437624    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwxk5\" (UniqueName: \"kubernetes.io/projected/35a5aa95-faba-4af6-a80e-137745490e73-kube-api-access-xwxk5\") pod \"maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl\" (UID: \"35a5aa95-faba-4af6-a80e-137745490e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"
Oct 30 21:30:00.441413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.437657    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052961e9-c81b-4273-94b9-4704b21dba4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-szvjv\" (UID: \"052961e9-c81b-4273-94b9-4704b21dba4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"
Oct 30 21:30:00.441413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.437751    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lltx4\" (UniqueName: \"kubernetes.io/projected/aca678bf-105c-472c-90e8-3226bdde7e25-kube-api-access-lltx4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25\" (UID: \"aca678bf-105c-472c-90e8-3226bdde7e25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"
Oct 30 21:30:00.441413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.437875    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcc2p\" (UniqueName: \"kubernetes.io/projected/052961e9-c81b-4273-94b9-4704b21dba4f-kube-api-access-mcc2p\") pod \"maas-api-key-cleanup-29889930-szvjv\" (UID: \"052961e9-c81b-4273-94b9-4704b21dba4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"
Oct 30 21:30:00.441413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.437940    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd73835-f5e2-47b8-8403-3aa7f03492eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7\" (UID: \"8fd73835-f5e2-47b8-8403-3aa7f03492eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"
Oct 30 21:30:00.441413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.437973    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e54b96-7b3a-421c-9ca0-312047e92c64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62\" (UID: \"49e54b96-7b3a-421c-9ca0-312047e92c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"
Oct 30 21:30:00.441413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.439011    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd73835-f5e2-47b8-8403-3aa7f03492eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7\" (UID: \"8fd73835-f5e2-47b8-8403-3aa7f03492eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.444054    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb8f64-ee03-4754-89ed-5c69010f4eba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c\" (UID: \"40bb8f64-ee03-4754-89ed-5c69010f4eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.444136    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/500bb411-3bf5-4e1a-bc15-36d332b0af13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl\" (UID: \"500bb411-3bf5-4e1a-bc15-36d332b0af13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.444178    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp27b\" (UniqueName: \"kubernetes.io/projected/500bb411-3bf5-4e1a-bc15-36d332b0af13-kube-api-access-qp27b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl\" (UID: \"500bb411-3bf5-4e1a-bc15-36d332b0af13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.444468    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca678bf-105c-472c-90e8-3226bdde7e25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25\" (UID: \"aca678bf-105c-472c-90e8-3226bdde7e25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.444528    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a5aa95-faba-4af6-a80e-137745490e73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl\" (UID: \"35a5aa95-faba-4af6-a80e-137745490e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.444586    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnjg\" (UniqueName: \"kubernetes.io/projected/8fd73835-f5e2-47b8-8403-3aa7f03492eb-kube-api-access-jvnjg\") pod \"maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7\" (UID: \"8fd73835-f5e2-47b8-8403-3aa7f03492eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.444653    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf881a3-86a2-494d-b8e5-3a789801bd41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp\" (UID: \"aaf881a3-86a2-494d-b8e5-3a789801bd41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.444721    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qf98\" (UniqueName: \"kubernetes.io/projected/aaf881a3-86a2-494d-b8e5-3a789801bd41-kube-api-access-4qf98\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp\" (UID: \"aaf881a3-86a2-494d-b8e5-3a789801bd41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.444771    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjcq\" (UniqueName: \"kubernetes.io/projected/40bb8f64-ee03-4754-89ed-5c69010f4eba-kube-api-access-ggjcq\") pod \"maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c\" (UID: \"40bb8f64-ee03-4754-89ed-5c69010f4eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.444953    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/500bb411-3bf5-4e1a-bc15-36d332b0af13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl\" (UID: \"500bb411-3bf5-4e1a-bc15-36d332b0af13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.445703    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf881a3-86a2-494d-b8e5-3a789801bd41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp\" (UID: \"aaf881a3-86a2-494d-b8e5-3a789801bd41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"
Oct 30 21:30:00.449971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.448103    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"]
Oct 30 21:30:00.464324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.462592    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qf98\" (UniqueName: \"kubernetes.io/projected/aaf881a3-86a2-494d-b8e5-3a789801bd41-kube-api-access-4qf98\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp\" (UID: \"aaf881a3-86a2-494d-b8e5-3a789801bd41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"
Oct 30 21:30:00.464324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.462789    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"]
Oct 30 21:30:00.464324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.462899    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"
Oct 30 21:30:00.471869 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.471836    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnjg\" (UniqueName: \"kubernetes.io/projected/8fd73835-f5e2-47b8-8403-3aa7f03492eb-kube-api-access-jvnjg\") pod \"maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7\" (UID: \"8fd73835-f5e2-47b8-8403-3aa7f03492eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"
Oct 30 21:30:00.473169 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.473018    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp27b\" (UniqueName: \"kubernetes.io/projected/500bb411-3bf5-4e1a-bc15-36d332b0af13-kube-api-access-qp27b\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl\" (UID: \"500bb411-3bf5-4e1a-bc15-36d332b0af13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"
Oct 30 21:30:00.474282 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.474055    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"
Oct 30 21:30:00.475551 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.475316    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"]
Oct 30 21:30:00.486952 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.486926    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"]
Oct 30 21:30:00.494716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.494519    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"]
Oct 30 21:30:00.494716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.494648    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"
Oct 30 21:30:00.507665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.507111    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"]
Oct 30 21:30:00.513749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.513730    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"]
Oct 30 21:30:00.521295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.521270    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"
Oct 30 21:30:00.523500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.523477    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"]
Oct 30 21:30:00.527270 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.527244    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"]
Oct 30 21:30:00.533646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.533624    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"
Oct 30 21:30:00.533959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.533938    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"
Oct 30 21:30:00.537200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.537155    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"]
Oct 30 21:30:00.545778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.545756    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb8f64-ee03-4754-89ed-5c69010f4eba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c\" (UID: \"40bb8f64-ee03-4754-89ed-5c69010f4eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"
Oct 30 21:30:00.545927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.545817    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89bsd\" (UniqueName: \"kubernetes.io/projected/daad11ef-c385-4cf8-975b-e63912445add-kube-api-access-89bsd\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h\" (UID: \"daad11ef-c385-4cf8-975b-e63912445add\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"
Oct 30 21:30:00.545927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.545918    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca678bf-105c-472c-90e8-3226bdde7e25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25\" (UID: \"aca678bf-105c-472c-90e8-3226bdde7e25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"
Oct 30 21:30:00.546113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.545963    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmr5\" (UniqueName: \"kubernetes.io/projected/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-kube-api-access-cbmr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p\" (UID: \"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"
Oct 30 21:30:00.546113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546011    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a5aa95-faba-4af6-a80e-137745490e73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl\" (UID: \"35a5aa95-faba-4af6-a80e-137745490e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"
Oct 30 21:30:00.546280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546147    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjcq\" (UniqueName: \"kubernetes.io/projected/40bb8f64-ee03-4754-89ed-5c69010f4eba-kube-api-access-ggjcq\") pod \"maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c\" (UID: \"40bb8f64-ee03-4754-89ed-5c69010f4eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"
Oct 30 21:30:00.546280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546206    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj6fn\" (UniqueName: \"kubernetes.io/projected/49e54b96-7b3a-421c-9ca0-312047e92c64-kube-api-access-zj6fn\") pod \"maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62\" (UID: \"49e54b96-7b3a-421c-9ca0-312047e92c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"
Oct 30 21:30:00.546280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546235    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad11ef-c385-4cf8-975b-e63912445add-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h\" (UID: \"daad11ef-c385-4cf8-975b-e63912445add\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"
Oct 30 21:30:00.546448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546285    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwxk5\" (UniqueName: \"kubernetes.io/projected/35a5aa95-faba-4af6-a80e-137745490e73-kube-api-access-xwxk5\") pod \"maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl\" (UID: \"35a5aa95-faba-4af6-a80e-137745490e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"
Oct 30 21:30:00.546448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546317    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052961e9-c81b-4273-94b9-4704b21dba4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-szvjv\" (UID: \"052961e9-c81b-4273-94b9-4704b21dba4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"
Oct 30 21:30:00.546448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546375    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnwxf\" (UniqueName: \"kubernetes.io/projected/64c97f25-107d-43c7-ad29-52fac553bedc-kube-api-access-fnwxf\") pod \"maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj\" (UID: \"64c97f25-107d-43c7-ad29-52fac553bedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"
Oct 30 21:30:00.546448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546412    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p\" (UID: \"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"
Oct 30 21:30:00.546671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546448    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx\" (UID: \"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"
Oct 30 21:30:00.546671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546476    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c97f25-107d-43c7-ad29-52fac553bedc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj\" (UID: \"64c97f25-107d-43c7-ad29-52fac553bedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"
Oct 30 21:30:00.546671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546518    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lltx4\" (UniqueName: \"kubernetes.io/projected/aca678bf-105c-472c-90e8-3226bdde7e25-kube-api-access-lltx4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25\" (UID: \"aca678bf-105c-472c-90e8-3226bdde7e25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"
Oct 30 21:30:00.546671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546555    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcc2p\" (UniqueName: \"kubernetes.io/projected/052961e9-c81b-4273-94b9-4704b21dba4f-kube-api-access-mcc2p\") pod \"maas-api-key-cleanup-29889930-szvjv\" (UID: \"052961e9-c81b-4273-94b9-4704b21dba4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"
Oct 30 21:30:00.546671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546600    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfjdn\" (UniqueName: \"kubernetes.io/projected/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-kube-api-access-gfjdn\") pod \"maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx\" (UID: \"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"
Oct 30 21:30:00.546671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.546645    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e54b96-7b3a-421c-9ca0-312047e92c64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62\" (UID: \"49e54b96-7b3a-421c-9ca0-312047e92c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"
Oct 30 21:30:00.547326 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.547304    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e54b96-7b3a-421c-9ca0-312047e92c64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62\" (UID: \"49e54b96-7b3a-421c-9ca0-312047e92c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"
Oct 30 21:30:00.547841 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.547820    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052961e9-c81b-4273-94b9-4704b21dba4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-szvjv\" (UID: \"052961e9-c81b-4273-94b9-4704b21dba4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"
Oct 30 21:30:00.548081 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.548052    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb8f64-ee03-4754-89ed-5c69010f4eba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c\" (UID: \"40bb8f64-ee03-4754-89ed-5c69010f4eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"
Oct 30 21:30:00.550049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.548620    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca678bf-105c-472c-90e8-3226bdde7e25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25\" (UID: \"aca678bf-105c-472c-90e8-3226bdde7e25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"
Oct 30 21:30:00.550049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.549262    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a5aa95-faba-4af6-a80e-137745490e73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl\" (UID: \"35a5aa95-faba-4af6-a80e-137745490e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"
Oct 30 21:30:00.557059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.557030    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lltx4\" (UniqueName: \"kubernetes.io/projected/aca678bf-105c-472c-90e8-3226bdde7e25-kube-api-access-lltx4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25\" (UID: \"aca678bf-105c-472c-90e8-3226bdde7e25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"
Oct 30 21:30:00.558415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.558392    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwxk5\" (UniqueName: \"kubernetes.io/projected/35a5aa95-faba-4af6-a80e-137745490e73-kube-api-access-xwxk5\") pod \"maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl\" (UID: \"35a5aa95-faba-4af6-a80e-137745490e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"
Oct 30 21:30:00.559030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.559010    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcc2p\" (UniqueName: \"kubernetes.io/projected/052961e9-c81b-4273-94b9-4704b21dba4f-kube-api-access-mcc2p\") pod \"maas-api-key-cleanup-29889930-szvjv\" (UID: \"052961e9-c81b-4273-94b9-4704b21dba4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"
Oct 30 21:30:00.564661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.564634    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj6fn\" (UniqueName: \"kubernetes.io/projected/49e54b96-7b3a-421c-9ca0-312047e92c64-kube-api-access-zj6fn\") pod \"maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62\" (UID: \"49e54b96-7b3a-421c-9ca0-312047e92c64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"
Oct 30 21:30:00.568346 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.568323    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjcq\" (UniqueName: \"kubernetes.io/projected/40bb8f64-ee03-4754-89ed-5c69010f4eba-kube-api-access-ggjcq\") pod \"maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c\" (UID: \"40bb8f64-ee03-4754-89ed-5c69010f4eba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"
Oct 30 21:30:00.602326 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.598277    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"
Oct 30 21:30:00.603213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.603189    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"
Oct 30 21:30:00.626090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.626059    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.648656    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjdn\" (UniqueName: \"kubernetes.io/projected/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-kube-api-access-gfjdn\") pod \"maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx\" (UID: \"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.648754    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89bsd\" (UniqueName: \"kubernetes.io/projected/daad11ef-c385-4cf8-975b-e63912445add-kube-api-access-89bsd\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h\" (UID: \"daad11ef-c385-4cf8-975b-e63912445add\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.648848    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmr5\" (UniqueName: \"kubernetes.io/projected/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-kube-api-access-cbmr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p\" (UID: \"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.649038    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv928\" (UniqueName: \"kubernetes.io/projected/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-kube-api-access-qv928\") pod \"maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx\" (UID: \"112b0e88-a621-4dc9-bf1f-1099de0cfe5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.649191    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad11ef-c385-4cf8-975b-e63912445add-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h\" (UID: \"daad11ef-c385-4cf8-975b-e63912445add\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.649235    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx\" (UID: \"112b0e88-a621-4dc9-bf1f-1099de0cfe5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.649304    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac65f6a-7beb-4c50-a0ba-546af2072612-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r\" (UID: \"1ac65f6a-7beb-4c50-a0ba-546af2072612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.649357    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnwxf\" (UniqueName: \"kubernetes.io/projected/64c97f25-107d-43c7-ad29-52fac553bedc-kube-api-access-fnwxf\") pod \"maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj\" (UID: \"64c97f25-107d-43c7-ad29-52fac553bedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.649839    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p\" (UID: \"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.649961    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx\" (UID: \"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.650000    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c97f25-107d-43c7-ad29-52fac553bedc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj\" (UID: \"64c97f25-107d-43c7-ad29-52fac553bedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.650054    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmff\" (UniqueName: \"kubernetes.io/projected/1ac65f6a-7beb-4c50-a0ba-546af2072612-kube-api-access-swmff\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r\" (UID: \"1ac65f6a-7beb-4c50-a0ba-546af2072612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.651518    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad11ef-c385-4cf8-975b-e63912445add-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h\" (UID: \"daad11ef-c385-4cf8-975b-e63912445add\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.652294    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p\" (UID: \"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.652992    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx\" (UID: \"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"
Oct 30 21:30:00.663738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.653645    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c97f25-107d-43c7-ad29-52fac553bedc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj\" (UID: \"64c97f25-107d-43c7-ad29-52fac553bedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"
Oct 30 21:30:00.665593 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.665561    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"
Oct 30 21:30:00.690047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.689814    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89bsd\" (UniqueName: \"kubernetes.io/projected/daad11ef-c385-4cf8-975b-e63912445add-kube-api-access-89bsd\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h\" (UID: \"daad11ef-c385-4cf8-975b-e63912445add\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"
Oct 30 21:30:00.692109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.692079    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmr5\" (UniqueName: \"kubernetes.io/projected/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-kube-api-access-cbmr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p\" (UID: \"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"
Oct 30 21:30:00.696452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.696401    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjdn\" (UniqueName: \"kubernetes.io/projected/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-kube-api-access-gfjdn\") pod \"maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx\" (UID: \"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"
Oct 30 21:30:00.700204 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.699627    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnwxf\" (UniqueName: \"kubernetes.io/projected/64c97f25-107d-43c7-ad29-52fac553bedc-kube-api-access-fnwxf\") pod \"maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj\" (UID: \"64c97f25-107d-43c7-ad29-52fac553bedc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"
Oct 30 21:30:00.710827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.710577    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"
Oct 30 21:30:00.742128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.739046    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"
Oct 30 21:30:00.743897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.743618    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"
Oct 30 21:30:00.753937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.752816    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv928\" (UniqueName: \"kubernetes.io/projected/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-kube-api-access-qv928\") pod \"maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx\" (UID: \"112b0e88-a621-4dc9-bf1f-1099de0cfe5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"
Oct 30 21:30:00.753937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.752998    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx\" (UID: \"112b0e88-a621-4dc9-bf1f-1099de0cfe5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"
Oct 30 21:30:00.753937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.753081    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac65f6a-7beb-4c50-a0ba-546af2072612-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r\" (UID: \"1ac65f6a-7beb-4c50-a0ba-546af2072612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"
Oct 30 21:30:00.753937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.753158    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmff\" (UniqueName: \"kubernetes.io/projected/1ac65f6a-7beb-4c50-a0ba-546af2072612-kube-api-access-swmff\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r\" (UID: \"1ac65f6a-7beb-4c50-a0ba-546af2072612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"
Oct 30 21:30:00.803468 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.755153    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac65f6a-7beb-4c50-a0ba-546af2072612-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r\" (UID: \"1ac65f6a-7beb-4c50-a0ba-546af2072612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"
Oct 30 21:30:00.803468 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.755712    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx\" (UID: \"112b0e88-a621-4dc9-bf1f-1099de0cfe5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"
Oct 30 21:30:00.804372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.804343    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"
Oct 30 21:30:00.804705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.804673    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"
Oct 30 21:30:00.835603 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.834267    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"
Oct 30 21:30:00.849208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.841651    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv928\" (UniqueName: \"kubernetes.io/projected/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-kube-api-access-qv928\") pod \"maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx\" (UID: \"112b0e88-a621-4dc9-bf1f-1099de0cfe5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"
Oct 30 21:30:00.849208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.845721    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"
Oct 30 21:30:00.849208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.849058    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmff\" (UniqueName: \"kubernetes.io/projected/1ac65f6a-7beb-4c50-a0ba-546af2072612-kube-api-access-swmff\") pod \"maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r\" (UID: \"1ac65f6a-7beb-4c50-a0ba-546af2072612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"
Oct 30 21:30:00.855631 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.855570    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"
Oct 30 21:30:00.946217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.945972    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"
Oct 30 21:30:00.965462 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:00.965233    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"
Oct 30 21:30:01.066600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.066335    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9" podUID="b92d5aab-dbcf-47e0-a2c9-111af9c207c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:01.070623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.070591    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"]
Oct 30 21:30:01.070780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.070742    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b" podUID="33b2eef0-0705-463e-9c89-96f86d0c1576" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:01.510885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.476274    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"]
Oct 30 21:30:01.510885 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:01.498098    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd73835_f5e2_47b8_8403_3aa7f03492eb.slice/crio-c9a685c901d0c352ba0fae4eba49b5a0c344754ee077c87a1afd432ef6d96d9e WatchSource:0}: Error finding container c9a685c901d0c352ba0fae4eba49b5a0c344754ee077c87a1afd432ef6d96d9e: Status 404 returned error can't find the container with id c9a685c901d0c352ba0fae4eba49b5a0c344754ee077c87a1afd432ef6d96d9e
Oct 30 21:30:01.736922 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:01.736892    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bb8f64_ee03_4754_89ed_5c69010f4eba.slice/crio-4bfe2c355bf947d97dbff9d4dfe157bc8f8c6d66cc519e1a1f428b2b3da3b3a4 WatchSource:0}: Error finding container 4bfe2c355bf947d97dbff9d4dfe157bc8f8c6d66cc519e1a1f428b2b3da3b3a4: Status 404 returned error can't find the container with id 4bfe2c355bf947d97dbff9d4dfe157bc8f8c6d66cc519e1a1f428b2b3da3b3a4
Oct 30 21:30:01.737093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.737062    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"]
Oct 30 21:30:01.757235 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:01.757202    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500bb411_3bf5_4e1a_bc15_36d332b0af13.slice/crio-1c925ff586a0bc807d2034b386370548941b395797adbb0dcce5ffa88d51635c WatchSource:0}: Error finding container 1c925ff586a0bc807d2034b386370548941b395797adbb0dcce5ffa88d51635c: Status 404 returned error can't find the container with id 1c925ff586a0bc807d2034b386370548941b395797adbb0dcce5ffa88d51635c
Oct 30 21:30:01.757626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.757596    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"]
Oct 30 21:30:01.818063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.815123    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"]
Oct 30 21:30:01.821061 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:01.821032    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf881a3_86a2_494d_b8e5_3a789801bd41.slice/crio-de3665607596fe36fa7b824f1d7cd8ca616ddb798969d13da4105061738699d4 WatchSource:0}: Error finding container de3665607596fe36fa7b824f1d7cd8ca616ddb798969d13da4105061738699d4: Status 404 returned error can't find the container with id de3665607596fe36fa7b824f1d7cd8ca616ddb798969d13da4105061738699d4
Oct 30 21:30:01.864428 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:01.864393    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca678bf_105c_472c_90e8_3226bdde7e25.slice/crio-cfec8f4e3323112992112e8c9b4ac908a3f95ad19a005ecfdcd4a428ad8730c4 WatchSource:0}: Error finding container cfec8f4e3323112992112e8c9b4ac908a3f95ad19a005ecfdcd4a428ad8730c4: Status 404 returned error can't find the container with id cfec8f4e3323112992112e8c9b4ac908a3f95ad19a005ecfdcd4a428ad8730c4
Oct 30 21:30:01.866651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.866614    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"]
Oct 30 21:30:01.892566 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.887017    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"]
Oct 30 21:30:01.899141 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.898603    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"]
Oct 30 21:30:01.968522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.968498    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"]
Oct 30 21:30:01.974741 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:01.974712    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"]
Oct 30 21:30:02.019006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.018975    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"]
Oct 30 21:30:02.027409 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.027370    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"]
Oct 30 21:30:02.028003 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:02.027961    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052961e9_c81b_4273_94b9_4704b21dba4f.slice/crio-8b20476e6a3c3ff11d4ca84026670197f999410e234068fde86e202c5d171a39 WatchSource:0}: Error finding container 8b20476e6a3c3ff11d4ca84026670197f999410e234068fde86e202c5d171a39: Status 404 returned error can't find the container with id 8b20476e6a3c3ff11d4ca84026670197f999410e234068fde86e202c5d171a39
Oct 30 21:30:02.031113 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:02.029944    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112b0e88_a621_4dc9_bf1f_1099de0cfe5d.slice/crio-9888451558d2508720f3ee6c6f2e2aca62ef5772e5f043b084d51a5fed20c7ab WatchSource:0}: Error finding container 9888451558d2508720f3ee6c6f2e2aca62ef5772e5f043b084d51a5fed20c7ab: Status 404 returned error can't find the container with id 9888451558d2508720f3ee6c6f2e2aca62ef5772e5f043b084d51a5fed20c7ab
Oct 30 21:30:02.036426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.036398    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"]
Oct 30 21:30:02.040261 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:02.040130    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaad11ef_c385_4cf8_975b_e63912445add.slice/crio-2102c8242d06999057a3f0baa84b6771408d20f70483274c0de027f54908b0af WatchSource:0}: Error finding container 2102c8242d06999057a3f0baa84b6771408d20f70483274c0de027f54908b0af: Status 404 returned error can't find the container with id 2102c8242d06999057a3f0baa84b6771408d20f70483274c0de027f54908b0af
Oct 30 21:30:02.065515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.065483    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"]
Oct 30 21:30:02.099983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.099960    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"]
Oct 30 21:30:02.212257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.212224    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" event={"ID":"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1","Type":"ContainerStarted","Data":"deb7209506b8d426d634f0a9dbc7f93a7180574d8f314f0c5f974170531a8014"}
Oct 30 21:30:02.214537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.214506    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" event={"ID":"7a4e9f94-c578-43a7-9f01-937d5cc8c679","Type":"ContainerStarted","Data":"ee4f731eccd59bb97a76edc9e23642560d07a9b410fa0ecce7cf896ef85aef6a"}
Oct 30 21:30:02.214666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.214557    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" event={"ID":"7a4e9f94-c578-43a7-9f01-937d5cc8c679","Type":"ContainerStarted","Data":"b6a8e2d632a54e19a3338f2f47f810610b78b922db01bab34f326b25004e842c"}
Oct 30 21:30:02.215943 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.215911    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" event={"ID":"35a5aa95-faba-4af6-a80e-137745490e73","Type":"ContainerStarted","Data":"d609d91381f16030ad875ee67f33b19542aad08787b1dc024a1a82c18c8b90a2"}
Oct 30 21:30:02.217233 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.217199    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj" event={"ID":"64c97f25-107d-43c7-ad29-52fac553bedc","Type":"ContainerStarted","Data":"41f0a2cf0c310d94527448c990312e2d34afb96f2a531ae58fb01624df43b020"}
Oct 30 21:30:02.218912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.218886    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv" event={"ID":"052961e9-c81b-4273-94b9-4704b21dba4f","Type":"ContainerStarted","Data":"8b20476e6a3c3ff11d4ca84026670197f999410e234068fde86e202c5d171a39"}
Oct 30 21:30:02.220337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.220304    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl" event={"ID":"500bb411-3bf5-4e1a-bc15-36d332b0af13","Type":"ContainerStarted","Data":"1c925ff586a0bc807d2034b386370548941b395797adbb0dcce5ffa88d51635c"}
Oct 30 21:30:02.221874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.221832    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" event={"ID":"112b0e88-a621-4dc9-bf1f-1099de0cfe5d","Type":"ContainerStarted","Data":"9888451558d2508720f3ee6c6f2e2aca62ef5772e5f043b084d51a5fed20c7ab"}
Oct 30 21:30:02.223407 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.223375    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h" event={"ID":"daad11ef-c385-4cf8-975b-e63912445add","Type":"ContainerStarted","Data":"2102c8242d06999057a3f0baa84b6771408d20f70483274c0de027f54908b0af"}
Oct 30 21:30:02.224808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.224786    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" event={"ID":"1ac65f6a-7beb-4c50-a0ba-546af2072612","Type":"ContainerStarted","Data":"8180e444bb4a5b32db17c8086355ad83c20294fd53112bc251db7d8f61f3e10e"}
Oct 30 21:30:02.226226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.226201    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c" event={"ID":"40bb8f64-ee03-4754-89ed-5c69010f4eba","Type":"ContainerStarted","Data":"4bfe2c355bf947d97dbff9d4dfe157bc8f8c6d66cc519e1a1f428b2b3da3b3a4"}
Oct 30 21:30:02.227887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.227832    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" event={"ID":"8fd73835-f5e2-47b8-8403-3aa7f03492eb","Type":"ContainerStarted","Data":"1ac64419cd1af037464c5fd3c27a3e8377d481d944bbcdb949efd239f61f7375"}
Oct 30 21:30:02.228000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.227900    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" event={"ID":"8fd73835-f5e2-47b8-8403-3aa7f03492eb","Type":"ContainerStarted","Data":"c9a685c901d0c352ba0fae4eba49b5a0c344754ee077c87a1afd432ef6d96d9e"}
Oct 30 21:30:02.229402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.229378    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25" event={"ID":"aca678bf-105c-472c-90e8-3226bdde7e25","Type":"ContainerStarted","Data":"cfec8f4e3323112992112e8c9b4ac908a3f95ad19a005ecfdcd4a428ad8730c4"}
Oct 30 21:30:02.230660 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.230619    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" podStartSLOduration=2.230609692 podStartE2EDuration="2.230609692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:02.229254079 +0000 UTC m=+3168.721070626" watchObservedRunningTime="2026-10-30 21:30:02.230609692 +0000 UTC m=+3168.722426236"
Oct 30 21:30:02.235249 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.235225    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" event={"ID":"aaf881a3-86a2-494d-b8e5-3a789801bd41","Type":"ContainerStarted","Data":"de3665607596fe36fa7b824f1d7cd8ca616ddb798969d13da4105061738699d4"}
Oct 30 21:30:02.250993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:02.250935    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" podStartSLOduration=2.250923813 podStartE2EDuration="2.250923813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:02.246081183 +0000 UTC m=+3168.737897728" watchObservedRunningTime="2026-10-30 21:30:02.250923813 +0000 UTC m=+3168.742740358"
Oct 30 21:30:02.379698 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:02.379661    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e54b96_7b3a_421c_9ca0_312047e92c64.slice/crio-77226071ca626eea89d3b35184bdb5a4721f7f4a93841c983ab9eb0f48368c92 WatchSource:0}: Error finding container 77226071ca626eea89d3b35184bdb5a4721f7f4a93841c983ab9eb0f48368c92: Status 404 returned error can't find the container with id 77226071ca626eea89d3b35184bdb5a4721f7f4a93841c983ab9eb0f48368c92
Oct 30 21:30:03.246422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.246387    4810 generic.go:356] "Generic (PLEG): container finished" podID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerID="b787114951da12d558db72660dc6385a551d0044642e786e06fb21a93618810d" exitCode=60
Oct 30 21:30:03.247254 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.246460    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl" event={"ID":"500bb411-3bf5-4e1a-bc15-36d332b0af13","Type":"ContainerDied","Data":"b787114951da12d558db72660dc6385a551d0044642e786e06fb21a93618810d"}
Oct 30 21:30:03.247254 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.246822    4810 scope.go:122] "RemoveContainer" containerID="b787114951da12d558db72660dc6385a551d0044642e786e06fb21a93618810d"
Oct 30 21:30:03.248958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.248788    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" event={"ID":"112b0e88-a621-4dc9-bf1f-1099de0cfe5d","Type":"ContainerStarted","Data":"37f6c563263dd9bd654ab83569d5ffe33f1dde35cb22a95fd9cdb0fa3f7ee6e2"}
Oct 30 21:30:03.250888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.250844    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62" event={"ID":"49e54b96-7b3a-421c-9ca0-312047e92c64","Type":"ContainerStarted","Data":"d178a7b2c6ef941634d45cb729b255eace9d69d68ce8e69e30f7ffe3a5d3a321"}
Oct 30 21:30:03.251066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.251048    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62" event={"ID":"49e54b96-7b3a-421c-9ca0-312047e92c64","Type":"ContainerStarted","Data":"77226071ca626eea89d3b35184bdb5a4721f7f4a93841c983ab9eb0f48368c92"}
Oct 30 21:30:03.252682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.252614    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" event={"ID":"1ac65f6a-7beb-4c50-a0ba-546af2072612","Type":"ContainerStarted","Data":"59c6cbfe411f37ad75308dbcca78b708fb11271048575302beab16c40410ac5b"}
Oct 30 21:30:03.254895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.254365    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c" event={"ID":"40bb8f64-ee03-4754-89ed-5c69010f4eba","Type":"ContainerStarted","Data":"d82454b38dc3ecc5c911dbed17eace9fb0db97279145d9ff53fe21eef737a877"}
Oct 30 21:30:03.256369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.256345    4810 generic.go:356] "Generic (PLEG): container finished" podID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerID="1ac64419cd1af037464c5fd3c27a3e8377d481d944bbcdb949efd239f61f7375" exitCode=60
Oct 30 21:30:03.256467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.256429    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" event={"ID":"8fd73835-f5e2-47b8-8403-3aa7f03492eb","Type":"ContainerDied","Data":"1ac64419cd1af037464c5fd3c27a3e8377d481d944bbcdb949efd239f61f7375"}
Oct 30 21:30:03.256653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.256635    4810 scope.go:122] "RemoveContainer" containerID="1ac64419cd1af037464c5fd3c27a3e8377d481d944bbcdb949efd239f61f7375"
Oct 30 21:30:03.258380 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.258359    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25" event={"ID":"aca678bf-105c-472c-90e8-3226bdde7e25","Type":"ContainerStarted","Data":"0f721ee6cc7ef67f78526118fa2687007c0d0958a8c9b4438d37ce8a72353df7"}
Oct 30 21:30:03.260097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.260077    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" event={"ID":"aaf881a3-86a2-494d-b8e5-3a789801bd41","Type":"ContainerStarted","Data":"9e5c80921cdd96bd1d9e7eb5e46d77a16563d512d32b9586e3a56dd78c1dfdbc"}
Oct 30 21:30:03.262672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.262646    4810 generic.go:356] "Generic (PLEG): container finished" podID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerID="745412982e7c72fb230c555984e50cf188f1fbf85d08ac0fe3b49d65a5eea380" exitCode=60
Oct 30 21:30:03.262767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.262709    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" event={"ID":"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1","Type":"ContainerDied","Data":"745412982e7c72fb230c555984e50cf188f1fbf85d08ac0fe3b49d65a5eea380"}
Oct 30 21:30:03.262967 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.262951    4810 scope.go:122] "RemoveContainer" containerID="745412982e7c72fb230c555984e50cf188f1fbf85d08ac0fe3b49d65a5eea380"
Oct 30 21:30:03.264500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.264451    4810 generic.go:356] "Generic (PLEG): container finished" podID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerID="ee4f731eccd59bb97a76edc9e23642560d07a9b410fa0ecce7cf896ef85aef6a" exitCode=60
Oct 30 21:30:03.264500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.264476    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" event={"ID":"7a4e9f94-c578-43a7-9f01-937d5cc8c679","Type":"ContainerDied","Data":"ee4f731eccd59bb97a76edc9e23642560d07a9b410fa0ecce7cf896ef85aef6a"}
Oct 30 21:30:03.264702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.264688    4810 scope.go:122] "RemoveContainer" containerID="ee4f731eccd59bb97a76edc9e23642560d07a9b410fa0ecce7cf896ef85aef6a"
Oct 30 21:30:03.265908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.265848    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p" event={"ID":"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380","Type":"ContainerStarted","Data":"523f70967f9ae7379807f79cdbfe34e7d4ba79fce3a8b1ff3bb6fad95f0367ad"}
Oct 30 21:30:03.267517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.267494    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" event={"ID":"35a5aa95-faba-4af6-a80e-137745490e73","Type":"ContainerStarted","Data":"72c2c247bc7bbba027851be7908050f6e26177c364fd9e59240109aacea726b1"}
Oct 30 21:30:03.269256 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.269117    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj" event={"ID":"64c97f25-107d-43c7-ad29-52fac553bedc","Type":"ContainerStarted","Data":"5efb1916f9ccde207f767322dd744c5ba9a79e3194dfeba49a194f83265f14de"}
Oct 30 21:30:03.299696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.299655    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv" event={"ID":"052961e9-c81b-4273-94b9-4704b21dba4f","Type":"ContainerStarted","Data":"1bc7273a08706bc291a14f5e211e0c23392c82c0ce5fc55c65806b1543cc21a0"}
Oct 30 21:30:03.388193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.388129    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" podStartSLOduration=3.388108343 podStartE2EDuration="3.388108343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.375220316 +0000 UTC m=+3169.867036904" watchObservedRunningTime="2026-10-30 21:30:03.388108343 +0000 UTC m=+3169.879924890"
Oct 30 21:30:03.399128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.399082    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" podStartSLOduration=3.3990686820000002 podStartE2EDuration="3.399068682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.396818904 +0000 UTC m=+3169.888635450" watchObservedRunningTime="2026-10-30 21:30:03.399068682 +0000 UTC m=+3169.890885221"
Oct 30 21:30:03.416184 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.416133    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62" podStartSLOduration=3.416118361 podStartE2EDuration="3.416118361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.416026961 +0000 UTC m=+3169.907843506" watchObservedRunningTime="2026-10-30 21:30:03.416118361 +0000 UTC m=+3169.907934900"
Oct 30 21:30:03.441265 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.441202    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25" podStartSLOduration=3.441182736 podStartE2EDuration="3.441182736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.434987145 +0000 UTC m=+3169.926803680" watchObservedRunningTime="2026-10-30 21:30:03.441182736 +0000 UTC m=+3169.932999283"
Oct 30 21:30:03.462514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.462460    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c" podStartSLOduration=3.462444408 podStartE2EDuration="3.462444408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.458404996 +0000 UTC m=+3169.950221543" watchObservedRunningTime="2026-10-30 21:30:03.462444408 +0000 UTC m=+3169.954260984"
Oct 30 21:30:03.499654 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.499591    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj" podStartSLOduration=3.499573843 podStartE2EDuration="3.499573843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.494982965 +0000 UTC m=+3169.986799511" watchObservedRunningTime="2026-10-30 21:30:03.499573843 +0000 UTC m=+3169.991390390"
Oct 30 21:30:03.517078 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.516971    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" podStartSLOduration=3.516951964 podStartE2EDuration="3.516951964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.516240504 +0000 UTC m=+3170.008057050" watchObservedRunningTime="2026-10-30 21:30:03.516951964 +0000 UTC m=+3170.008768510"
Oct 30 21:30:03.538198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.538145    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" podStartSLOduration=3.538129076 podStartE2EDuration="3.538129076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.534334121 +0000 UTC m=+3170.026150660" watchObservedRunningTime="2026-10-30 21:30:03.538129076 +0000 UTC m=+3170.029945621"
Oct 30 21:30:03.554362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:03.554305    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv" podStartSLOduration=3.554288839 podStartE2EDuration="3.554288839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.550762878 +0000 UTC m=+3170.042579423" watchObservedRunningTime="2026-10-30 21:30:03.554288839 +0000 UTC m=+3170.046105383"
Oct 30 21:30:04.311894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.311843    4810 generic.go:356] "Generic (PLEG): container finished" podID="64c97f25-107d-43c7-ad29-52fac553bedc" containerID="5efb1916f9ccde207f767322dd744c5ba9a79e3194dfeba49a194f83265f14de" exitCode=60
Oct 30 21:30:04.312459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.311928    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj" event={"ID":"64c97f25-107d-43c7-ad29-52fac553bedc","Type":"ContainerDied","Data":"5efb1916f9ccde207f767322dd744c5ba9a79e3194dfeba49a194f83265f14de"}
Oct 30 21:30:04.312459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.312292    4810 scope.go:122] "RemoveContainer" containerID="5efb1916f9ccde207f767322dd744c5ba9a79e3194dfeba49a194f83265f14de"
Oct 30 21:30:04.314037 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.314015    4810 generic.go:356] "Generic (PLEG): container finished" podID="052961e9-c81b-4273-94b9-4704b21dba4f" containerID="1bc7273a08706bc291a14f5e211e0c23392c82c0ce5fc55c65806b1543cc21a0" exitCode=60
Oct 30 21:30:04.314174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.314103    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv" event={"ID":"052961e9-c81b-4273-94b9-4704b21dba4f","Type":"ContainerDied","Data":"1bc7273a08706bc291a14f5e211e0c23392c82c0ce5fc55c65806b1543cc21a0"}
Oct 30 21:30:04.314379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.314328    4810 scope.go:122] "RemoveContainer" containerID="1bc7273a08706bc291a14f5e211e0c23392c82c0ce5fc55c65806b1543cc21a0"
Oct 30 21:30:04.339336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.338923    4810 generic.go:356] "Generic (PLEG): container finished" podID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerID="37f6c563263dd9bd654ab83569d5ffe33f1dde35cb22a95fd9cdb0fa3f7ee6e2" exitCode=60
Oct 30 21:30:04.339336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.339217    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" event={"ID":"112b0e88-a621-4dc9-bf1f-1099de0cfe5d","Type":"ContainerDied","Data":"37f6c563263dd9bd654ab83569d5ffe33f1dde35cb22a95fd9cdb0fa3f7ee6e2"}
Oct 30 21:30:04.339693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.339639    4810 scope.go:122] "RemoveContainer" containerID="37f6c563263dd9bd654ab83569d5ffe33f1dde35cb22a95fd9cdb0fa3f7ee6e2"
Oct 30 21:30:04.346271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.345631    4810 generic.go:356] "Generic (PLEG): container finished" podID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerID="d178a7b2c6ef941634d45cb729b255eace9d69d68ce8e69e30f7ffe3a5d3a321" exitCode=60
Oct 30 21:30:04.346271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.345775    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62" event={"ID":"49e54b96-7b3a-421c-9ca0-312047e92c64","Type":"ContainerDied","Data":"d178a7b2c6ef941634d45cb729b255eace9d69d68ce8e69e30f7ffe3a5d3a321"}
Oct 30 21:30:04.346271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.346226    4810 scope.go:122] "RemoveContainer" containerID="d178a7b2c6ef941634d45cb729b255eace9d69d68ce8e69e30f7ffe3a5d3a321"
Oct 30 21:30:04.350998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.350974    4810 generic.go:356] "Generic (PLEG): container finished" podID="daad11ef-c385-4cf8-975b-e63912445add" containerID="1b5fa5dad846b4960ad7cc53a65b7ac8568b40caa727517acf4fa6138fdf9435" exitCode=60
Oct 30 21:30:04.351136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.351053    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h" event={"ID":"daad11ef-c385-4cf8-975b-e63912445add","Type":"ContainerDied","Data":"1b5fa5dad846b4960ad7cc53a65b7ac8568b40caa727517acf4fa6138fdf9435"}
Oct 30 21:30:04.351383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.351365    4810 scope.go:122] "RemoveContainer" containerID="1b5fa5dad846b4960ad7cc53a65b7ac8568b40caa727517acf4fa6138fdf9435"
Oct 30 21:30:04.352938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.352916    4810 generic.go:356] "Generic (PLEG): container finished" podID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerID="59c6cbfe411f37ad75308dbcca78b708fb11271048575302beab16c40410ac5b" exitCode=60
Oct 30 21:30:04.353029 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.353005    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" event={"ID":"1ac65f6a-7beb-4c50-a0ba-546af2072612","Type":"ContainerDied","Data":"59c6cbfe411f37ad75308dbcca78b708fb11271048575302beab16c40410ac5b"}
Oct 30 21:30:04.353397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.353376    4810 scope.go:122] "RemoveContainer" containerID="59c6cbfe411f37ad75308dbcca78b708fb11271048575302beab16c40410ac5b"
Oct 30 21:30:04.355912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.355887    4810 generic.go:356] "Generic (PLEG): container finished" podID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerID="d82454b38dc3ecc5c911dbed17eace9fb0db97279145d9ff53fe21eef737a877" exitCode=60
Oct 30 21:30:04.356025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.356003    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c" event={"ID":"40bb8f64-ee03-4754-89ed-5c69010f4eba","Type":"ContainerDied","Data":"d82454b38dc3ecc5c911dbed17eace9fb0db97279145d9ff53fe21eef737a877"}
Oct 30 21:30:04.356380 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.356360    4810 scope.go:122] "RemoveContainer" containerID="d82454b38dc3ecc5c911dbed17eace9fb0db97279145d9ff53fe21eef737a877"
Oct 30 21:30:04.358103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.358084    4810 generic.go:356] "Generic (PLEG): container finished" podID="aca678bf-105c-472c-90e8-3226bdde7e25" containerID="0f721ee6cc7ef67f78526118fa2687007c0d0958a8c9b4438d37ce8a72353df7" exitCode=60
Oct 30 21:30:04.358200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.358146    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25" event={"ID":"aca678bf-105c-472c-90e8-3226bdde7e25","Type":"ContainerDied","Data":"0f721ee6cc7ef67f78526118fa2687007c0d0958a8c9b4438d37ce8a72353df7"}
Oct 30 21:30:04.358383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.358366    4810 scope.go:122] "RemoveContainer" containerID="0f721ee6cc7ef67f78526118fa2687007c0d0958a8c9b4438d37ce8a72353df7"
Oct 30 21:30:04.359848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.359829    4810 generic.go:356] "Generic (PLEG): container finished" podID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerID="9e5c80921cdd96bd1d9e7eb5e46d77a16563d512d32b9586e3a56dd78c1dfdbc" exitCode=60
Oct 30 21:30:04.360011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.359915    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" event={"ID":"aaf881a3-86a2-494d-b8e5-3a789801bd41","Type":"ContainerDied","Data":"9e5c80921cdd96bd1d9e7eb5e46d77a16563d512d32b9586e3a56dd78c1dfdbc"}
Oct 30 21:30:04.360117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.360100    4810 scope.go:122] "RemoveContainer" containerID="9e5c80921cdd96bd1d9e7eb5e46d77a16563d512d32b9586e3a56dd78c1dfdbc"
Oct 30 21:30:04.361650 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.361631    4810 generic.go:356] "Generic (PLEG): container finished" podID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerID="026fd1a4599afea1b0b1378eab590bc4dc3db3ae96b832d163c9c71ad1b77d30" exitCode=60
Oct 30 21:30:04.361765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.361708    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p" event={"ID":"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380","Type":"ContainerDied","Data":"026fd1a4599afea1b0b1378eab590bc4dc3db3ae96b832d163c9c71ad1b77d30"}
Oct 30 21:30:04.361894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.361876    4810 scope.go:122] "RemoveContainer" containerID="026fd1a4599afea1b0b1378eab590bc4dc3db3ae96b832d163c9c71ad1b77d30"
Oct 30 21:30:04.363259 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.363240    4810 generic.go:356] "Generic (PLEG): container finished" podID="35a5aa95-faba-4af6-a80e-137745490e73" containerID="72c2c247bc7bbba027851be7908050f6e26177c364fd9e59240109aacea726b1" exitCode=60
Oct 30 21:30:04.363350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.363305    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" event={"ID":"35a5aa95-faba-4af6-a80e-137745490e73","Type":"ContainerDied","Data":"72c2c247bc7bbba027851be7908050f6e26177c364fd9e59240109aacea726b1"}
Oct 30 21:30:04.363470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:04.363457    4810 scope.go:122] "RemoveContainer" containerID="72c2c247bc7bbba027851be7908050f6e26177c364fd9e59240109aacea726b1"
Oct 30 21:30:05.381182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.380992    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl" event={"ID":"500bb411-3bf5-4e1a-bc15-36d332b0af13","Type":"ContainerStarted","Data":"8693b32576c63472238c146696565272b4d53048a810b99060cde6eb24908fa4"}
Oct 30 21:30:05.398776 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.398741    4810 generic.go:356] "Generic (PLEG): container finished" podID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerID="f5cf98748bad58007a2c4a569e3b1071684d405b09a95b58218ce5b67b3f3046" exitCode=60
Oct 30 21:30:05.398966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.398891    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" event={"ID":"8fd73835-f5e2-47b8-8403-3aa7f03492eb","Type":"ContainerDied","Data":"f5cf98748bad58007a2c4a569e3b1071684d405b09a95b58218ce5b67b3f3046"}
Oct 30 21:30:05.398966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.398950    4810 scope.go:122] "RemoveContainer" containerID="1ac64419cd1af037464c5fd3c27a3e8377d481d944bbcdb949efd239f61f7375"
Oct 30 21:30:05.399322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.399300    4810 scope.go:122] "RemoveContainer" containerID="f5cf98748bad58007a2c4a569e3b1071684d405b09a95b58218ce5b67b3f3046"
Oct 30 21:30:05.399833 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:05.399729    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7_odh-ai-gateway-infra(8fd73835-f5e2-47b8-8403-3aa7f03492eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb"
Oct 30 21:30:05.406844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.406819    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" event={"ID":"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1","Type":"ContainerStarted","Data":"5cd627d7a9d5639088d95625fb2ee1e101a21a9477dd379dc92a7dfec890aea9"}
Oct 30 21:30:05.408441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.407545    4810 scope.go:122] "RemoveContainer" containerID="5cd627d7a9d5639088d95625fb2ee1e101a21a9477dd379dc92a7dfec890aea9"
Oct 30 21:30:05.409169 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:05.409143    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx_odh-ai-gateway-infra(cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1"
Oct 30 21:30:05.414373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.414293    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl" podStartSLOduration=5.414274911 podStartE2EDuration="5.414274911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:05.41138214 +0000 UTC m=+3171.903198687" watchObservedRunningTime="2026-10-30 21:30:05.414274911 +0000 UTC m=+3171.906091457"
Oct 30 21:30:05.421241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.420469    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" event={"ID":"7a4e9f94-c578-43a7-9f01-937d5cc8c679","Type":"ContainerStarted","Data":"11ae15836672cb4d24c288a84ecc7222c61b954720f4292f038922a29bc27afd"}
Oct 30 21:30:05.421241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.420816    4810 scope.go:122] "RemoveContainer" containerID="11ae15836672cb4d24c288a84ecc7222c61b954720f4292f038922a29bc27afd"
Oct 30 21:30:05.421241 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:05.421197    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css_odh-ai-gateway-infra(7a4e9f94-c578-43a7-9f01-937d5cc8c679)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679"
Oct 30 21:30:05.739513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.739477    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"]
Oct 30 21:30:05.751945 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:05.751411    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"]
Oct 30 21:30:06.458214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.458161    4810 generic.go:356] "Generic (PLEG): container finished" podID="64c97f25-107d-43c7-ad29-52fac553bedc" containerID="1b750e3b1321355eeb2b10432a5967fc1d716832017dbebcf2d8a2f2b069264a" exitCode=60
Oct 30 21:30:06.458752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.458290    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj" event={"ID":"64c97f25-107d-43c7-ad29-52fac553bedc","Type":"ContainerDied","Data":"1b750e3b1321355eeb2b10432a5967fc1d716832017dbebcf2d8a2f2b069264a"}
Oct 30 21:30:06.458752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.458385    4810 scope.go:122] "RemoveContainer" containerID="5efb1916f9ccde207f767322dd744c5ba9a79e3194dfeba49a194f83265f14de"
Oct 30 21:30:06.461696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.461662    4810 scope.go:122] "RemoveContainer" containerID="1b750e3b1321355eeb2b10432a5967fc1d716832017dbebcf2d8a2f2b069264a"
Oct 30 21:30:06.462668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.462642    4810 generic.go:356] "Generic (PLEG): container finished" podID="052961e9-c81b-4273-94b9-4704b21dba4f" containerID="09ee7cfbf8a154efae72771b6d81b3e8a5fd67a42695e11566e91cd5397f1b3b" exitCode=60
Oct 30 21:30:06.462742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.462672    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv" event={"ID":"052961e9-c81b-4273-94b9-4704b21dba4f","Type":"ContainerDied","Data":"09ee7cfbf8a154efae72771b6d81b3e8a5fd67a42695e11566e91cd5397f1b3b"}
Oct 30 21:30:06.462742 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.462619    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj_odh-ai-gateway-infra(64c97f25-107d-43c7-ad29-52fac553bedc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj" podUID="64c97f25-107d-43c7-ad29-52fac553bedc"
Oct 30 21:30:06.463179 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.463154    4810 scope.go:122] "RemoveContainer" containerID="09ee7cfbf8a154efae72771b6d81b3e8a5fd67a42695e11566e91cd5397f1b3b"
Oct 30 21:30:06.463734 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.463705    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-szvjv_odh-ai-gateway-infra(052961e9-c81b-4273-94b9-4704b21dba4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv" podUID="052961e9-c81b-4273-94b9-4704b21dba4f"
Oct 30 21:30:06.467095 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.465815    4810 generic.go:356] "Generic (PLEG): container finished" podID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerID="8693b32576c63472238c146696565272b4d53048a810b99060cde6eb24908fa4" exitCode=60
Oct 30 21:30:06.467095 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.465994    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl" event={"ID":"500bb411-3bf5-4e1a-bc15-36d332b0af13","Type":"ContainerDied","Data":"8693b32576c63472238c146696565272b4d53048a810b99060cde6eb24908fa4"}
Oct 30 21:30:06.467095 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.466740    4810 scope.go:122] "RemoveContainer" containerID="8693b32576c63472238c146696565272b4d53048a810b99060cde6eb24908fa4"
Oct 30 21:30:06.467095 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.467030    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl_odh-ai-gateway-infra(500bb411-3bf5-4e1a-bc15-36d332b0af13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13"
Oct 30 21:30:06.469063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.469043    4810 generic.go:356] "Generic (PLEG): container finished" podID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerID="a3195a969e93ca8b5ea51d015661a564c83c8bfb30a5d37d6b4fcdc4eca7fcb1" exitCode=60
Oct 30 21:30:06.469149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.469102    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" event={"ID":"112b0e88-a621-4dc9-bf1f-1099de0cfe5d","Type":"ContainerDied","Data":"a3195a969e93ca8b5ea51d015661a564c83c8bfb30a5d37d6b4fcdc4eca7fcb1"}
Oct 30 21:30:06.469474 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.469363    4810 scope.go:122] "RemoveContainer" containerID="a3195a969e93ca8b5ea51d015661a564c83c8bfb30a5d37d6b4fcdc4eca7fcb1"
Oct 30 21:30:06.469594 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.469575    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx_odh-ai-gateway-infra(112b0e88-a621-4dc9-bf1f-1099de0cfe5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d"
Oct 30 21:30:06.471773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.471751    4810 generic.go:356] "Generic (PLEG): container finished" podID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerID="b04032598659c1205037a51e328afbedb6515ef9d1c12c9ba89ca157b499a470" exitCode=60
Oct 30 21:30:06.471900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.471831    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62" event={"ID":"49e54b96-7b3a-421c-9ca0-312047e92c64","Type":"ContainerDied","Data":"b04032598659c1205037a51e328afbedb6515ef9d1c12c9ba89ca157b499a470"}
Oct 30 21:30:06.472071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.472056    4810 scope.go:122] "RemoveContainer" containerID="b04032598659c1205037a51e328afbedb6515ef9d1c12c9ba89ca157b499a470"
Oct 30 21:30:06.472309 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.472284    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62_odh-ai-gateway-infra(49e54b96-7b3a-421c-9ca0-312047e92c64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64"
Oct 30 21:30:06.473894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.473874    4810 generic.go:356] "Generic (PLEG): container finished" podID="daad11ef-c385-4cf8-975b-e63912445add" containerID="6204d4ef75cf584cf72ffe2164af6844c1d67cc6014f693f42b0782ae4c89c12" exitCode=60
Oct 30 21:30:06.473985 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.473951    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h" event={"ID":"daad11ef-c385-4cf8-975b-e63912445add","Type":"ContainerDied","Data":"6204d4ef75cf584cf72ffe2164af6844c1d67cc6014f693f42b0782ae4c89c12"}
Oct 30 21:30:06.474228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.474207    4810 scope.go:122] "RemoveContainer" containerID="6204d4ef75cf584cf72ffe2164af6844c1d67cc6014f693f42b0782ae4c89c12"
Oct 30 21:30:06.474581 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.474443    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h_odh-ai-gateway-infra(daad11ef-c385-4cf8-975b-e63912445add)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h" podUID="daad11ef-c385-4cf8-975b-e63912445add"
Oct 30 21:30:06.476366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.476343    4810 generic.go:356] "Generic (PLEG): container finished" podID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerID="37211fa0765cc80d3e1a9304545a1273a88902b91390dc994dac058706cddf5c" exitCode=60
Oct 30 21:30:06.476459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.476421    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" event={"ID":"1ac65f6a-7beb-4c50-a0ba-546af2072612","Type":"ContainerDied","Data":"37211fa0765cc80d3e1a9304545a1273a88902b91390dc994dac058706cddf5c"}
Oct 30 21:30:06.476704 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.476689    4810 scope.go:122] "RemoveContainer" containerID="37211fa0765cc80d3e1a9304545a1273a88902b91390dc994dac058706cddf5c"
Oct 30 21:30:06.476939 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.476914    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r_odh-ai-gateway-infra(1ac65f6a-7beb-4c50-a0ba-546af2072612)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612"
Oct 30 21:30:06.479673 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.479650    4810 generic.go:356] "Generic (PLEG): container finished" podID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerID="f2d8f91119244a3918b966d3627d8f9795d5e6114894461006c8594beb7ed87f" exitCode=60
Oct 30 21:30:06.479769 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.479751    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c" event={"ID":"40bb8f64-ee03-4754-89ed-5c69010f4eba","Type":"ContainerDied","Data":"f2d8f91119244a3918b966d3627d8f9795d5e6114894461006c8594beb7ed87f"}
Oct 30 21:30:06.480101 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.480086    4810 scope.go:122] "RemoveContainer" containerID="f2d8f91119244a3918b966d3627d8f9795d5e6114894461006c8594beb7ed87f"
Oct 30 21:30:06.480304 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.480285    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c_odh-ai-gateway-infra(40bb8f64-ee03-4754-89ed-5c69010f4eba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba"
Oct 30 21:30:06.482274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.481950    4810 scope.go:122] "RemoveContainer" containerID="f5cf98748bad58007a2c4a569e3b1071684d405b09a95b58218ce5b67b3f3046"
Oct 30 21:30:06.482274 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.482234    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7_odh-ai-gateway-infra(8fd73835-f5e2-47b8-8403-3aa7f03492eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb"
Oct 30 21:30:06.483983 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.483959    4810 generic.go:356] "Generic (PLEG): container finished" podID="aca678bf-105c-472c-90e8-3226bdde7e25" containerID="1ad66ef0024e4b3dbac652719a710edaa1b80ec51afc1b597f2d8daec197b125" exitCode=60
Oct 30 21:30:06.484075 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.484035    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25" event={"ID":"aca678bf-105c-472c-90e8-3226bdde7e25","Type":"ContainerDied","Data":"1ad66ef0024e4b3dbac652719a710edaa1b80ec51afc1b597f2d8daec197b125"}
Oct 30 21:30:06.484564 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.484542    4810 scope.go:122] "RemoveContainer" containerID="1ad66ef0024e4b3dbac652719a710edaa1b80ec51afc1b597f2d8daec197b125"
Oct 30 21:30:06.484779 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.484757    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25_odh-ai-gateway-infra(aca678bf-105c-472c-90e8-3226bdde7e25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25" podUID="aca678bf-105c-472c-90e8-3226bdde7e25"
Oct 30 21:30:06.486130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.486106    4810 generic.go:356] "Generic (PLEG): container finished" podID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerID="453678314e52b76c6ebc8ca8a359487755a4433532c543eb9b30b62be3a55ff3" exitCode=60
Oct 30 21:30:06.486243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.486178    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" event={"ID":"aaf881a3-86a2-494d-b8e5-3a789801bd41","Type":"ContainerDied","Data":"453678314e52b76c6ebc8ca8a359487755a4433532c543eb9b30b62be3a55ff3"}
Oct 30 21:30:06.486396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.486380    4810 scope.go:122] "RemoveContainer" containerID="453678314e52b76c6ebc8ca8a359487755a4433532c543eb9b30b62be3a55ff3"
Oct 30 21:30:06.486652 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.486611    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp_odh-ai-gateway-infra(aaf881a3-86a2-494d-b8e5-3a789801bd41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41"
Oct 30 21:30:06.488238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.488179    4810 generic.go:356] "Generic (PLEG): container finished" podID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerID="5cd627d7a9d5639088d95625fb2ee1e101a21a9477dd379dc92a7dfec890aea9" exitCode=60
Oct 30 21:30:06.488341 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.488241    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" event={"ID":"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1","Type":"ContainerDied","Data":"5cd627d7a9d5639088d95625fb2ee1e101a21a9477dd379dc92a7dfec890aea9"}
Oct 30 21:30:06.488463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.488449    4810 scope.go:122] "RemoveContainer" containerID="5cd627d7a9d5639088d95625fb2ee1e101a21a9477dd379dc92a7dfec890aea9"
Oct 30 21:30:06.488664 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.488646    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx_odh-ai-gateway-infra(cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1"
Oct 30 21:30:06.490385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.490366    4810 generic.go:356] "Generic (PLEG): container finished" podID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerID="11ae15836672cb4d24c288a84ecc7222c61b954720f4292f038922a29bc27afd" exitCode=60
Oct 30 21:30:06.490485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.490431    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" event={"ID":"7a4e9f94-c578-43a7-9f01-937d5cc8c679","Type":"ContainerDied","Data":"11ae15836672cb4d24c288a84ecc7222c61b954720f4292f038922a29bc27afd"}
Oct 30 21:30:06.490631 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.490618    4810 scope.go:122] "RemoveContainer" containerID="11ae15836672cb4d24c288a84ecc7222c61b954720f4292f038922a29bc27afd"
Oct 30 21:30:06.490886 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.490840    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css_odh-ai-gateway-infra(7a4e9f94-c578-43a7-9f01-937d5cc8c679)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679"
Oct 30 21:30:06.492485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.492464    4810 generic.go:356] "Generic (PLEG): container finished" podID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerID="e843271db4acd069e603933378531896f404bd6dfef528a860f4602f88959b0f" exitCode=60
Oct 30 21:30:06.492582 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.492536    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p" event={"ID":"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380","Type":"ContainerDied","Data":"e843271db4acd069e603933378531896f404bd6dfef528a860f4602f88959b0f"}
Oct 30 21:30:06.492720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.492706    4810 scope.go:122] "RemoveContainer" containerID="e843271db4acd069e603933378531896f404bd6dfef528a860f4602f88959b0f"
Oct 30 21:30:06.492939 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.492918    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p_odh-ai-gateway-infra(cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380"
Oct 30 21:30:06.494808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.494769    4810 generic.go:356] "Generic (PLEG): container finished" podID="35a5aa95-faba-4af6-a80e-137745490e73" containerID="c117983d292dd7b04235c1e521c62750a86a6ef4a758355a51282a5f3a585073" exitCode=60
Oct 30 21:30:06.494958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.494838    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" event={"ID":"35a5aa95-faba-4af6-a80e-137745490e73","Type":"ContainerDied","Data":"c117983d292dd7b04235c1e521c62750a86a6ef4a758355a51282a5f3a585073"}
Oct 30 21:30:06.495155 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.495131    4810 scope.go:122] "RemoveContainer" containerID="c117983d292dd7b04235c1e521c62750a86a6ef4a758355a51282a5f3a585073"
Oct 30 21:30:06.495340 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:06.495316    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl_odh-ai-gateway-infra(35a5aa95-faba-4af6-a80e-137745490e73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" podUID="35a5aa95-faba-4af6-a80e-137745490e73"
Oct 30 21:30:06.647010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.646986    4810 scope.go:122] "RemoveContainer" containerID="1bc7273a08706bc291a14f5e211e0c23392c82c0ce5fc55c65806b1543cc21a0"
Oct 30 21:30:06.662723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.662699    4810 scope.go:122] "RemoveContainer" containerID="b787114951da12d558db72660dc6385a551d0044642e786e06fb21a93618810d"
Oct 30 21:30:06.686940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.686912    4810 scope.go:122] "RemoveContainer" containerID="37f6c563263dd9bd654ab83569d5ffe33f1dde35cb22a95fd9cdb0fa3f7ee6e2"
Oct 30 21:30:06.704901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.704549    4810 scope.go:122] "RemoveContainer" containerID="d178a7b2c6ef941634d45cb729b255eace9d69d68ce8e69e30f7ffe3a5d3a321"
Oct 30 21:30:06.722389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.722363    4810 scope.go:122] "RemoveContainer" containerID="1b5fa5dad846b4960ad7cc53a65b7ac8568b40caa727517acf4fa6138fdf9435"
Oct 30 21:30:06.738460 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.738438    4810 scope.go:122] "RemoveContainer" containerID="59c6cbfe411f37ad75308dbcca78b708fb11271048575302beab16c40410ac5b"
Oct 30 21:30:06.754332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.754306    4810 scope.go:122] "RemoveContainer" containerID="d82454b38dc3ecc5c911dbed17eace9fb0db97279145d9ff53fe21eef737a877"
Oct 30 21:30:06.774831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.774621    4810 scope.go:122] "RemoveContainer" containerID="0f721ee6cc7ef67f78526118fa2687007c0d0958a8c9b4438d37ce8a72353df7"
Oct 30 21:30:06.793468 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.793446    4810 scope.go:122] "RemoveContainer" containerID="9e5c80921cdd96bd1d9e7eb5e46d77a16563d512d32b9586e3a56dd78c1dfdbc"
Oct 30 21:30:06.809678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.809650    4810 scope.go:122] "RemoveContainer" containerID="745412982e7c72fb230c555984e50cf188f1fbf85d08ac0fe3b49d65a5eea380"
Oct 30 21:30:06.825387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.825363    4810 scope.go:122] "RemoveContainer" containerID="ee4f731eccd59bb97a76edc9e23642560d07a9b410fa0ecce7cf896ef85aef6a"
Oct 30 21:30:06.840377 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.840350    4810 scope.go:122] "RemoveContainer" containerID="026fd1a4599afea1b0b1378eab590bc4dc3db3ae96b832d163c9c71ad1b77d30"
Oct 30 21:30:06.856413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:06.856389    4810 scope.go:122] "RemoveContainer" containerID="72c2c247bc7bbba027851be7908050f6e26177c364fd9e59240109aacea726b1"
Oct 30 21:30:07.508880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.508202    4810 scope.go:122] "RemoveContainer" containerID="09ee7cfbf8a154efae72771b6d81b3e8a5fd67a42695e11566e91cd5397f1b3b"
Oct 30 21:30:07.508880 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.508620    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-szvjv_odh-ai-gateway-infra(052961e9-c81b-4273-94b9-4704b21dba4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv" podUID="052961e9-c81b-4273-94b9-4704b21dba4f"
Oct 30 21:30:07.510221 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.510195    4810 scope.go:122] "RemoveContainer" containerID="8693b32576c63472238c146696565272b4d53048a810b99060cde6eb24908fa4"
Oct 30 21:30:07.510558 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.510532    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl_odh-ai-gateway-infra(500bb411-3bf5-4e1a-bc15-36d332b0af13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13"
Oct 30 21:30:07.512030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.512014    4810 scope.go:122] "RemoveContainer" containerID="a3195a969e93ca8b5ea51d015661a564c83c8bfb30a5d37d6b4fcdc4eca7fcb1"
Oct 30 21:30:07.512293 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.512265    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx_odh-ai-gateway-infra(112b0e88-a621-4dc9-bf1f-1099de0cfe5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d"
Oct 30 21:30:07.514080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.514063    4810 scope.go:122] "RemoveContainer" containerID="b04032598659c1205037a51e328afbedb6515ef9d1c12c9ba89ca157b499a470"
Oct 30 21:30:07.514337 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.514320    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62_odh-ai-gateway-infra(49e54b96-7b3a-421c-9ca0-312047e92c64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64"
Oct 30 21:30:07.516038 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.516017    4810 scope.go:122] "RemoveContainer" containerID="6204d4ef75cf584cf72ffe2164af6844c1d67cc6014f693f42b0782ae4c89c12"
Oct 30 21:30:07.516306 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.516285    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h_odh-ai-gateway-infra(daad11ef-c385-4cf8-975b-e63912445add)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h" podUID="daad11ef-c385-4cf8-975b-e63912445add"
Oct 30 21:30:07.517760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.517730    4810 scope.go:122] "RemoveContainer" containerID="37211fa0765cc80d3e1a9304545a1273a88902b91390dc994dac058706cddf5c"
Oct 30 21:30:07.518022 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.517993    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r_odh-ai-gateway-infra(1ac65f6a-7beb-4c50-a0ba-546af2072612)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612"
Oct 30 21:30:07.519431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.519412    4810 scope.go:122] "RemoveContainer" containerID="f2d8f91119244a3918b966d3627d8f9795d5e6114894461006c8594beb7ed87f"
Oct 30 21:30:07.519639 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.519619    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c_odh-ai-gateway-infra(40bb8f64-ee03-4754-89ed-5c69010f4eba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba"
Oct 30 21:30:07.521229 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.521212    4810 scope.go:122] "RemoveContainer" containerID="1ad66ef0024e4b3dbac652719a710edaa1b80ec51afc1b597f2d8daec197b125"
Oct 30 21:30:07.521445 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.521420    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25_odh-ai-gateway-infra(aca678bf-105c-472c-90e8-3226bdde7e25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25" podUID="aca678bf-105c-472c-90e8-3226bdde7e25"
Oct 30 21:30:07.523044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.523022    4810 scope.go:122] "RemoveContainer" containerID="453678314e52b76c6ebc8ca8a359487755a4433532c543eb9b30b62be3a55ff3"
Oct 30 21:30:07.523286 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.523264    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp_odh-ai-gateway-infra(aaf881a3-86a2-494d-b8e5-3a789801bd41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41"
Oct 30 21:30:07.524792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.524775    4810 scope.go:122] "RemoveContainer" containerID="5cd627d7a9d5639088d95625fb2ee1e101a21a9477dd379dc92a7dfec890aea9"
Oct 30 21:30:07.525053 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.525027    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx_odh-ai-gateway-infra(cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1"
Oct 30 21:30:07.526958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.526933    4810 scope.go:122] "RemoveContainer" containerID="11ae15836672cb4d24c288a84ecc7222c61b954720f4292f038922a29bc27afd"
Oct 30 21:30:07.527196 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.527171    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css_odh-ai-gateway-infra(7a4e9f94-c578-43a7-9f01-937d5cc8c679)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679"
Oct 30 21:30:07.529012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.528992    4810 scope.go:122] "RemoveContainer" containerID="e843271db4acd069e603933378531896f404bd6dfef528a860f4602f88959b0f"
Oct 30 21:30:07.529268 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.529245    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p_odh-ai-gateway-infra(cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380"
Oct 30 21:30:07.530758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.530741    4810 scope.go:122] "RemoveContainer" containerID="c117983d292dd7b04235c1e521c62750a86a6ef4a758355a51282a5f3a585073"
Oct 30 21:30:07.531010 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.530991    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl_odh-ai-gateway-infra(35a5aa95-faba-4af6-a80e-137745490e73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" podUID="35a5aa95-faba-4af6-a80e-137745490e73"
Oct 30 21:30:07.532716 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:07.532696    4810 scope.go:122] "RemoveContainer" containerID="1b750e3b1321355eeb2b10432a5967fc1d716832017dbebcf2d8a2f2b069264a"
Oct 30 21:30:07.533014 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:07.532992    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj_odh-ai-gateway-infra(64c97f25-107d-43c7-ad29-52fac553bedc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj" podUID="64c97f25-107d-43c7-ad29-52fac553bedc"
Oct 30 21:30:08.872567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:08.870171    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"]
Oct 30 21:30:08.872567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:08.870426    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk" podUID="63a2b983-ff5b-411f-85de-01a6dbae639e" containerName="maas-api" containerID="cri-o://fbeb6975061ca04e46a841d9fb643c5dfab272896319e2c635461b0ee54856ea" gracePeriod=30
Oct 30 21:30:09.528569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.528500    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"]
Oct 30 21:30:09.537810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.537770    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffce13-btkpq"]
Oct 30 21:30:09.638835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.638797    4810 generic.go:356] "Generic (PLEG): container finished" podID="63a2b983-ff5b-411f-85de-01a6dbae639e" containerID="fbeb6975061ca04e46a841d9fb643c5dfab272896319e2c635461b0ee54856ea" exitCode=0
Oct 30 21:30:09.639099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.639071    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk" event={"ID":"63a2b983-ff5b-411f-85de-01a6dbae639e","Type":"ContainerDied","Data":"fbeb6975061ca04e46a841d9fb643c5dfab272896319e2c635461b0ee54856ea"}
Oct 30 21:30:09.670970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.670942    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:30:09.672091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.672066    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8062a4d5-5fc5-4c4e-949d-f74659ff6ba9" path="/var/lib/kubelet/pods/8062a4d5-5fc5-4c4e-949d-f74659ff6ba9/volumes"
Oct 30 21:30:09.830248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.830160    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a2b983-ff5b-411f-85de-01a6dbae639e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a2b983-ff5b-411f-85de-01a6dbae639e-openshift-service-ca\") pod \"63a2b983-ff5b-411f-85de-01a6dbae639e\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") "
Oct 30 21:30:09.830420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.830249    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-maas-api-tls\") pod \"63a2b983-ff5b-411f-85de-01a6dbae639e\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") "
Oct 30 21:30:09.830420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.830272    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-metrics-certs\") pod \"63a2b983-ff5b-411f-85de-01a6dbae639e\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") "
Oct 30 21:30:09.830420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.830328    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a2b983-ff5b-411f-85de-01a6dbae639e-kube-api-access-z46p6\" (UniqueName: \"kubernetes.io/projected/63a2b983-ff5b-411f-85de-01a6dbae639e-kube-api-access-z46p6\") pod \"63a2b983-ff5b-411f-85de-01a6dbae639e\" (UID: \"63a2b983-ff5b-411f-85de-01a6dbae639e\") "
Oct 30 21:30:09.830664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.830637    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a2b983-ff5b-411f-85de-01a6dbae639e-openshift-service-ca" pod "63a2b983-ff5b-411f-85de-01a6dbae639e" (UID: "63a2b983-ff5b-411f-85de-01a6dbae639e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:09.831850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.831104    4810 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a2b983-ff5b-411f-85de-01a6dbae639e-openshift-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:09.832707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.832684    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-maas-api-tls" pod "63a2b983-ff5b-411f-85de-01a6dbae639e" (UID: "63a2b983-ff5b-411f-85de-01a6dbae639e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:09.832799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.832731    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-metrics-certs" pod "63a2b983-ff5b-411f-85de-01a6dbae639e" (UID: "63a2b983-ff5b-411f-85de-01a6dbae639e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:09.833012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.832997    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a2b983-ff5b-411f-85de-01a6dbae639e-kube-api-access-z46p6" pod "63a2b983-ff5b-411f-85de-01a6dbae639e" (UID: "63a2b983-ff5b-411f-85de-01a6dbae639e"). InnerVolumeSpecName "kube-api-access-z46p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:09.932781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.932743    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-maas-api-tls\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:09.932781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.932777    4810 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63a2b983-ff5b-411f-85de-01a6dbae639e-metrics-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:09.933245 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:09.932791    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z46p6\" (UniqueName: \"kubernetes.io/projected/63a2b983-ff5b-411f-85de-01a6dbae639e-kube-api-access-z46p6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:10.649972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:10.649941    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"
Oct 30 21:30:10.650209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:10.649972    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk" event={"ID":"63a2b983-ff5b-411f-85de-01a6dbae639e","Type":"ContainerDied","Data":"3d5d66349b70f7793efe1ff84fd0fe8d3db01f5e132fa8dc2bfe6b3e6d9667a0"}
Oct 30 21:30:10.650209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:10.650042    4810 scope.go:122] "RemoveContainer" containerID="fbeb6975061ca04e46a841d9fb643c5dfab272896319e2c635461b0ee54856ea"
Oct 30 21:30:10.750881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:10.747314    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"]
Oct 30 21:30:10.750881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:10.747310    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6" podUID="6b22d659-57de-4142-8da1-8b0705a58fc0" containerName="payload-processing" containerID="cri-o://b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc" gracePeriod=25
Oct 30 21:30:10.752458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:10.752419    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8" podUID="5b2866f7-5408-4815-b3b3-c898ce73181f" containerName="payload-pre-processing" containerID="cri-o://b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4" gracePeriod=25
Oct 30 21:30:10.752774 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:10.752750    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffce13-545bdc5d8b-sskbk"]
Oct 30 21:30:11.059594 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.059560    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:30:11.060490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.060460    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b" podUID="33b2eef0-0705-463e-9c89-96f86d0c1576" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:11.060490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.060482    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9" podUID="b92d5aab-dbcf-47e0-a2c9-111af9c207c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:11.154655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.154619    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b22d659-57de-4142-8da1-8b0705a58fc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b22d659-57de-4142-8da1-8b0705a58fc0-plugins-config-volume\") pod \"6b22d659-57de-4142-8da1-8b0705a58fc0\" (UID: \"6b22d659-57de-4142-8da1-8b0705a58fc0\") "
Oct 30 21:30:11.154834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.154775    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b22d659-57de-4142-8da1-8b0705a58fc0-kube-api-access-mc2kf\" (UniqueName: \"kubernetes.io/projected/6b22d659-57de-4142-8da1-8b0705a58fc0-kube-api-access-mc2kf\") pod \"6b22d659-57de-4142-8da1-8b0705a58fc0\" (UID: \"6b22d659-57de-4142-8da1-8b0705a58fc0\") "
Oct 30 21:30:11.155084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.155057    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b22d659-57de-4142-8da1-8b0705a58fc0-plugins-config-volume" pod "6b22d659-57de-4142-8da1-8b0705a58fc0" (UID: "6b22d659-57de-4142-8da1-8b0705a58fc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:11.155338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.155318    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b22d659-57de-4142-8da1-8b0705a58fc0-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:11.156964 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.156941    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b22d659-57de-4142-8da1-8b0705a58fc0-kube-api-access-mc2kf" pod "6b22d659-57de-4142-8da1-8b0705a58fc0" (UID: "6b22d659-57de-4142-8da1-8b0705a58fc0"). InnerVolumeSpecName "kube-api-access-mc2kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:11.257575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.257450    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc2kf\" (UniqueName: \"kubernetes.io/projected/6b22d659-57de-4142-8da1-8b0705a58fc0-kube-api-access-mc2kf\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:11.637435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.637392    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"]
Oct 30 21:30:11.729836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.729784    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a2b983-ff5b-411f-85de-01a6dbae639e" path="/var/lib/kubelet/pods/63a2b983-ff5b-411f-85de-01a6dbae639e/volumes"
Oct 30 21:30:11.731057 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.731021    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2ac526-plskw"]
Oct 30 21:30:11.761792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.761752    4810 generic.go:356] "Generic (PLEG): container finished" podID="6b22d659-57de-4142-8da1-8b0705a58fc0" containerID="b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc" exitCode=0
Oct 30 21:30:11.762184 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.762160    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6" event={"ID":"6b22d659-57de-4142-8da1-8b0705a58fc0","Type":"ContainerDied","Data":"b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc"}
Oct 30 21:30:11.762267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.762253    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6" event={"ID":"6b22d659-57de-4142-8da1-8b0705a58fc0","Type":"ContainerDied","Data":"75aef960c55f1a1cbee0eccb1ad0e5704f0fb4e58193a5d950d02fde0a6b7c96"}
Oct 30 21:30:11.762327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.762292    4810 scope.go:122] "RemoveContainer" containerID="b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc"
Oct 30 21:30:11.762583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.762569    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"
Oct 30 21:30:11.798989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.798953    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"]
Oct 30 21:30:11.810781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.810749    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2ac526-5b545557d7-zw4r6"]
Oct 30 21:30:11.815028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.815000    4810 scope.go:122] "RemoveContainer" containerID="b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc"
Oct 30 21:30:11.815670 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:11.815637    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc\": container with ID starting with b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc not found: ID does not exist" containerID="b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc"
Oct 30 21:30:11.815780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:11.815668    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc"} err="failed to get container status \"b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc\": rpc error: code = NotFound desc = could not find container \"b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc\": container with ID starting with b07da3c6548a07dde069a9df8635af990c6f73f71a370f9a9dab47695d81e0bc not found: ID does not exist"
Oct 30 21:30:11.830405 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:11.830377    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b22d659_57de_4142_8da1_8b0705a58fc0.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:30:12.781193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.779906    4810 generic.go:356] "Generic (PLEG): container finished" podID="33b2eef0-0705-463e-9c89-96f86d0c1576" containerID="9d0d3b76b844914770f7f46c2d70f855f9595098e2fb5ede719791adb78b802d" exitCode=0
Oct 30 21:30:12.781193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.779982    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b" event={"ID":"33b2eef0-0705-463e-9c89-96f86d0c1576","Type":"ContainerDied","Data":"9d0d3b76b844914770f7f46c2d70f855f9595098e2fb5ede719791adb78b802d"}
Oct 30 21:30:12.781193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.780047    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b" event={"ID":"33b2eef0-0705-463e-9c89-96f86d0c1576","Type":"ContainerDied","Data":"7421d7c0bdbe68c15c5bb91f6d332515d94357bc1d4c6bdae85b8af7ab16bf3d"}
Oct 30 21:30:12.781193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.780070    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7421d7c0bdbe68c15c5bb91f6d332515d94357bc1d4c6bdae85b8af7ab16bf3d"
Oct 30 21:30:12.786627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.783980    4810 generic.go:356] "Generic (PLEG): container finished" podID="b92d5aab-dbcf-47e0-a2c9-111af9c207c6" containerID="743d59e793a1835ec8fe8945d8604c54269fb7db392888046a60c9190a182a46" exitCode=0
Oct 30 21:30:12.786627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.784024    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9" event={"ID":"b92d5aab-dbcf-47e0-a2c9-111af9c207c6","Type":"ContainerDied","Data":"743d59e793a1835ec8fe8945d8604c54269fb7db392888046a60c9190a182a46"}
Oct 30 21:30:12.802963 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.801350    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:30:12.834701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.833840    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b2eef0-0705-463e-9c89-96f86d0c1576-kube-api-access-fvkdk\" (UniqueName: \"kubernetes.io/projected/33b2eef0-0705-463e-9c89-96f86d0c1576-kube-api-access-fvkdk\") pod \"33b2eef0-0705-463e-9c89-96f86d0c1576\" (UID: \"33b2eef0-0705-463e-9c89-96f86d0c1576\") "
Oct 30 21:30:12.834701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.833957    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b2eef0-0705-463e-9c89-96f86d0c1576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b2eef0-0705-463e-9c89-96f86d0c1576-plugins-config-volume\") pod \"33b2eef0-0705-463e-9c89-96f86d0c1576\" (UID: \"33b2eef0-0705-463e-9c89-96f86d0c1576\") "
Oct 30 21:30:12.834701 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.834697    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b2eef0-0705-463e-9c89-96f86d0c1576-plugins-config-volume" pod "33b2eef0-0705-463e-9c89-96f86d0c1576" (UID: "33b2eef0-0705-463e-9c89-96f86d0c1576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:12.836666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.836637    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b2eef0-0705-463e-9c89-96f86d0c1576-kube-api-access-fvkdk" pod "33b2eef0-0705-463e-9c89-96f86d0c1576" (UID: "33b2eef0-0705-463e-9c89-96f86d0c1576"). InnerVolumeSpecName "kube-api-access-fvkdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:12.863207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.863184    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:30:12.935601 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.935567    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-kube-api-access-2fvrs\" (UniqueName: \"kubernetes.io/projected/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-kube-api-access-2fvrs\") pod \"b92d5aab-dbcf-47e0-a2c9-111af9c207c6\" (UID: \"b92d5aab-dbcf-47e0-a2c9-111af9c207c6\") "
Oct 30 21:30:12.935795 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.935637    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-plugins-config-volume\") pod \"b92d5aab-dbcf-47e0-a2c9-111af9c207c6\" (UID: \"b92d5aab-dbcf-47e0-a2c9-111af9c207c6\") "
Oct 30 21:30:12.936044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.936021    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-plugins-config-volume" pod "b92d5aab-dbcf-47e0-a2c9-111af9c207c6" (UID: "b92d5aab-dbcf-47e0-a2c9-111af9c207c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:12.936378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.936356    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvkdk\" (UniqueName: \"kubernetes.io/projected/33b2eef0-0705-463e-9c89-96f86d0c1576-kube-api-access-fvkdk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:12.936429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.936386    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b2eef0-0705-463e-9c89-96f86d0c1576-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:12.936429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.936402    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:12.937767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:12.937747    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-kube-api-access-2fvrs" pod "b92d5aab-dbcf-47e0-a2c9-111af9c207c6" (UID: "b92d5aab-dbcf-47e0-a2c9-111af9c207c6"). InnerVolumeSpecName "kube-api-access-2fvrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:13.038093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.038015    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fvrs\" (UniqueName: \"kubernetes.io/projected/b92d5aab-dbcf-47e0-a2c9-111af9c207c6-kube-api-access-2fvrs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:13.664717 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.664687    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b22d659-57de-4142-8da1-8b0705a58fc0" path="/var/lib/kubelet/pods/6b22d659-57de-4142-8da1-8b0705a58fc0/volumes"
Oct 30 21:30:13.665160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.665145    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0a0666-bb04-4253-b9a9-bf15716f3db6" path="/var/lib/kubelet/pods/eb0a0666-bb04-4253-b9a9-bf15716f3db6/volumes"
Oct 30 21:30:13.793908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.793810    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"
Oct 30 21:30:13.793908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.793806    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9" event={"ID":"b92d5aab-dbcf-47e0-a2c9-111af9c207c6","Type":"ContainerDied","Data":"f877e7ced65022f2bcbee54e16350afc545f58c48b409e3084f1c9dd90424bf7"}
Oct 30 21:30:13.794483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.793955    4810 scope.go:122] "RemoveContainer" containerID="743d59e793a1835ec8fe8945d8604c54269fb7db392888046a60c9190a182a46"
Oct 30 21:30:13.794483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.794015    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"
Oct 30 21:30:13.821210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.821178    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"]
Oct 30 21:30:13.825653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.825630    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffce13-7bc988d7cf-s4bv9"]
Oct 30 21:30:13.837499 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.837471    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"]
Oct 30 21:30:13.843478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:13.843456    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffce13-b86df9f45-tvl4b"]
Oct 30 21:30:15.683889 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:15.683821    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b2eef0-0705-463e-9c89-96f86d0c1576" path="/var/lib/kubelet/pods/33b2eef0-0705-463e-9c89-96f86d0c1576/volumes"
Oct 30 21:30:15.684618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:15.684594    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92d5aab-dbcf-47e0-a2c9-111af9c207c6" path="/var/lib/kubelet/pods/b92d5aab-dbcf-47e0-a2c9-111af9c207c6/volumes"
Oct 30 21:30:16.655473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:16.655437    4810 scope.go:122] "RemoveContainer" containerID="11ae15836672cb4d24c288a84ecc7222c61b954720f4292f038922a29bc27afd"
Oct 30 21:30:17.655293 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:17.655258    4810 scope.go:122] "RemoveContainer" containerID="8693b32576c63472238c146696565272b4d53048a810b99060cde6eb24908fa4"
Oct 30 21:30:17.656369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:17.656343    4810 scope.go:122] "RemoveContainer" containerID="09ee7cfbf8a154efae72771b6d81b3e8a5fd67a42695e11566e91cd5397f1b3b"
Oct 30 21:30:17.657416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:17.657400    4810 scope.go:122] "RemoveContainer" containerID="c117983d292dd7b04235c1e521c62750a86a6ef4a758355a51282a5f3a585073"
Oct 30 21:30:17.860017 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:17.859976    4810 generic.go:356] "Generic (PLEG): container finished" podID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerID="f031e3e7e46c6b9da248974156a79b26b908bd1799dbba943c2975dc90b038ac" exitCode=60
Oct 30 21:30:17.860185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:17.860039    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" event={"ID":"7a4e9f94-c578-43a7-9f01-937d5cc8c679","Type":"ContainerDied","Data":"f031e3e7e46c6b9da248974156a79b26b908bd1799dbba943c2975dc90b038ac"}
Oct 30 21:30:17.860185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:17.860105    4810 scope.go:122] "RemoveContainer" containerID="11ae15836672cb4d24c288a84ecc7222c61b954720f4292f038922a29bc27afd"
Oct 30 21:30:17.860529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:17.860511    4810 scope.go:122] "RemoveContainer" containerID="f031e3e7e46c6b9da248974156a79b26b908bd1799dbba943c2975dc90b038ac"
Oct 30 21:30:17.860934 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:17.860905    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css_odh-ai-gateway-infra(7a4e9f94-c578-43a7-9f01-937d5cc8c679)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679"
Oct 30 21:30:18.655322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.655286    4810 scope.go:122] "RemoveContainer" containerID="5cd627d7a9d5639088d95625fb2ee1e101a21a9477dd379dc92a7dfec890aea9"
Oct 30 21:30:18.655826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.655442    4810 scope.go:122] "RemoveContainer" containerID="453678314e52b76c6ebc8ca8a359487755a4433532c543eb9b30b62be3a55ff3"
Oct 30 21:30:18.655826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.655588    4810 scope.go:122] "RemoveContainer" containerID="b04032598659c1205037a51e328afbedb6515ef9d1c12c9ba89ca157b499a470"
Oct 30 21:30:18.655826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.655698    4810 scope.go:122] "RemoveContainer" containerID="6204d4ef75cf584cf72ffe2164af6844c1d67cc6014f693f42b0782ae4c89c12"
Oct 30 21:30:18.847309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.847259    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8" podUID="5b2866f7-5408-4815-b3b3-c898ce73181f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:18.888240 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.888192    4810 generic.go:356] "Generic (PLEG): container finished" podID="052961e9-c81b-4273-94b9-4704b21dba4f" containerID="8eb477b2797a7786057df2e533e54b82c151dad2ed1d5227965d2ec6cdb00b25" exitCode=60
Oct 30 21:30:18.888403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.888337    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv" event={"ID":"052961e9-c81b-4273-94b9-4704b21dba4f","Type":"ContainerDied","Data":"8eb477b2797a7786057df2e533e54b82c151dad2ed1d5227965d2ec6cdb00b25"}
Oct 30 21:30:18.888468 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.888426    4810 scope.go:122] "RemoveContainer" containerID="09ee7cfbf8a154efae72771b6d81b3e8a5fd67a42695e11566e91cd5397f1b3b"
Oct 30 21:30:18.890125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.890105    4810 scope.go:122] "RemoveContainer" containerID="8eb477b2797a7786057df2e533e54b82c151dad2ed1d5227965d2ec6cdb00b25"
Oct 30 21:30:18.890633 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:18.890484    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-szvjv_odh-ai-gateway-infra(052961e9-c81b-4273-94b9-4704b21dba4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv" podUID="052961e9-c81b-4273-94b9-4704b21dba4f"
Oct 30 21:30:18.892669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.892641    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl" event={"ID":"500bb411-3bf5-4e1a-bc15-36d332b0af13","Type":"ContainerStarted","Data":"e00432a425cbdbd29259983d9aec53c4a78f79fb309112d18c113c55ec779a97"}
Oct 30 21:30:18.893371 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.893347    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"]
Oct 30 21:30:18.898320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.898276    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" event={"ID":"35a5aa95-faba-4af6-a80e-137745490e73","Type":"ContainerStarted","Data":"981c95d94f41d7ac88f09c6050ff7aed393ffef3a572bf06c91c505446362c6f"}
Oct 30 21:30:18.898676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:18.898662    4810 scope.go:122] "RemoveContainer" containerID="981c95d94f41d7ac88f09c6050ff7aed393ffef3a572bf06c91c505446362c6f"
Oct 30 21:30:18.899028 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:18.899010    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl_odh-ai-gateway-infra(35a5aa95-faba-4af6-a80e-137745490e73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" podUID="35a5aa95-faba-4af6-a80e-137745490e73"
Oct 30 21:30:19.140873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.140839    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"
Oct 30 21:30:19.205950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.205898    4810 scope.go:122] "RemoveContainer" containerID="6e3565375bb61481193cbd901134d8693eadbc049c922a4e7e56e1b109150990"
Oct 30 21:30:19.225825 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.225708    4810 scope.go:122] "RemoveContainer" containerID="57d2f5ab6fcf01abd9019d7262abe657aa0c87321d8eb07952431a9f3ff191ad"
Oct 30 21:30:19.238148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.238126    4810 scope.go:122] "RemoveContainer" containerID="9d0d3b76b844914770f7f46c2d70f855f9595098e2fb5ede719791adb78b802d"
Oct 30 21:30:19.238610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.238585    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a4e9f94-c578-43a7-9f01-937d5cc8c679-kube-api-access-q66qt\" (UniqueName: \"kubernetes.io/projected/7a4e9f94-c578-43a7-9f01-937d5cc8c679-kube-api-access-q66qt\") pod \"7a4e9f94-c578-43a7-9f01-937d5cc8c679\" (UID: \"7a4e9f94-c578-43a7-9f01-937d5cc8c679\") "
Oct 30 21:30:19.238751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.238652    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a4e9f94-c578-43a7-9f01-937d5cc8c679-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a4e9f94-c578-43a7-9f01-937d5cc8c679-maas-api-service-ca\") pod \"7a4e9f94-c578-43a7-9f01-937d5cc8c679\" (UID: \"7a4e9f94-c578-43a7-9f01-937d5cc8c679\") "
Oct 30 21:30:19.239308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.239285    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a4e9f94-c578-43a7-9f01-937d5cc8c679-maas-api-service-ca" pod "7a4e9f94-c578-43a7-9f01-937d5cc8c679" (UID: "7a4e9f94-c578-43a7-9f01-937d5cc8c679"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:19.239415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.239398    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a4e9f94-c578-43a7-9f01-937d5cc8c679-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:19.241129 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.241108    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4e9f94-c578-43a7-9f01-937d5cc8c679-kube-api-access-q66qt" pod "7a4e9f94-c578-43a7-9f01-937d5cc8c679" (UID: "7a4e9f94-c578-43a7-9f01-937d5cc8c679"). InnerVolumeSpecName "kube-api-access-q66qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:19.249458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.249437    4810 scope.go:122] "RemoveContainer" containerID="d79df4cb045aa5ed767723a79ca545f139d07ab8a3724d4747ccb6f31903d5f9"
Oct 30 21:30:19.259829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.259810    4810 scope.go:122] "RemoveContainer" containerID="4bc227132ba6a3df5265584d1dcbfef136c4d10a94e1f6b434fd3a31947664c0"
Oct 30 21:30:19.341296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.341269    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q66qt\" (UniqueName: \"kubernetes.io/projected/7a4e9f94-c578-43a7-9f01-937d5cc8c679-kube-api-access-q66qt\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:19.588759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.588668    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"]
Oct 30 21:30:19.588993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.588942    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy" containerID="cri-o://98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e" gracePeriod=30
Oct 30 21:30:19.656416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.655731    4810 scope.go:122] "RemoveContainer" containerID="a3195a969e93ca8b5ea51d015661a564c83c8bfb30a5d37d6b4fcdc4eca7fcb1"
Oct 30 21:30:19.658850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.657206    4810 scope.go:122] "RemoveContainer" containerID="37211fa0765cc80d3e1a9304545a1273a88902b91390dc994dac058706cddf5c"
Oct 30 21:30:19.659427 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.659400    4810 scope.go:122] "RemoveContainer" containerID="1ad66ef0024e4b3dbac652719a710edaa1b80ec51afc1b597f2d8daec197b125"
Oct 30 21:30:19.660498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.660473    4810 scope.go:122] "RemoveContainer" containerID="e843271db4acd069e603933378531896f404bd6dfef528a860f4602f88959b0f"
Oct 30 21:30:19.664441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.661175    4810 scope.go:122] "RemoveContainer" containerID="f5cf98748bad58007a2c4a569e3b1071684d405b09a95b58218ce5b67b3f3046"
Oct 30 21:30:19.700220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.700195    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"]
Oct 30 21:30:19.993785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.993760    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"]
Oct 30 21:30:19.998362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:19.998340    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"]
Oct 30 21:30:20.017379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.013164    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"]
Oct 30 21:30:20.027634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.027137    4810 generic.go:356] "Generic (PLEG): container finished" podID="35a5aa95-faba-4af6-a80e-137745490e73" containerID="981c95d94f41d7ac88f09c6050ff7aed393ffef3a572bf06c91c505446362c6f" exitCode=60
Oct 30 21:30:20.027634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.027358    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" event={"ID":"35a5aa95-faba-4af6-a80e-137745490e73","Type":"ContainerDied","Data":"981c95d94f41d7ac88f09c6050ff7aed393ffef3a572bf06c91c505446362c6f"}
Oct 30 21:30:20.027634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.027425    4810 scope.go:122] "RemoveContainer" containerID="c117983d292dd7b04235c1e521c62750a86a6ef4a758355a51282a5f3a585073"
Oct 30 21:30:20.037948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.037619    4810 generic.go:356] "Generic (PLEG): container finished" podID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerID="e00432a425cbdbd29259983d9aec53c4a78f79fb309112d18c113c55ec779a97" exitCode=60
Oct 30 21:30:20.037948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.037711    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl" event={"ID":"500bb411-3bf5-4e1a-bc15-36d332b0af13","Type":"ContainerDied","Data":"e00432a425cbdbd29259983d9aec53c4a78f79fb309112d18c113c55ec779a97"}
Oct 30 21:30:20.047595 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.047531    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62" event={"ID":"49e54b96-7b3a-421c-9ca0-312047e92c64","Type":"ContainerStarted","Data":"7c083e82ea583dcfb529a5890d827a189d924e2ccab282e432e7a457052e82c6"}
Oct 30 21:30:20.081397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.081179    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h" event={"ID":"daad11ef-c385-4cf8-975b-e63912445add","Type":"ContainerStarted","Data":"d8474e468c30f9565ff52cd0602a6edbca80f420d078054d3972b2c60f53a4e2"}
Oct 30 21:30:20.087053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.087013    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" event={"ID":"aaf881a3-86a2-494d-b8e5-3a789801bd41","Type":"ContainerStarted","Data":"cfad3f4a43d9fc505c7eb9d0c85044a5ef1bcddc30c303584bf5ebb3190d168b"}
Oct 30 21:30:20.090069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.089920    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" event={"ID":"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1","Type":"ContainerStarted","Data":"1053669f93d4ce15b5e66a357d9fe29ab4ec79f2d78904e7877e1ec779d37eea"}
Oct 30 21:30:20.096322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.094151    4810 scope.go:122] "RemoveContainer" containerID="cfad3f4a43d9fc505c7eb9d0c85044a5ef1bcddc30c303584bf5ebb3190d168b"
Oct 30 21:30:20.096322 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:20.094451    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp_odh-ai-gateway-infra(aaf881a3-86a2-494d-b8e5-3a789801bd41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41"
Oct 30 21:30:20.105503 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.104808    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"
Oct 30 21:30:20.108966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.105773    4810 scope.go:122] "RemoveContainer" containerID="d8474e468c30f9565ff52cd0602a6edbca80f420d078054d3972b2c60f53a4e2"
Oct 30 21:30:20.108966 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:20.106151    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h_odh-ai-gateway-infra(daad11ef-c385-4cf8-975b-e63912445add)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h" podUID="daad11ef-c385-4cf8-975b-e63912445add"
Oct 30 21:30:20.108966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.107074    4810 scope.go:122] "RemoveContainer" containerID="1053669f93d4ce15b5e66a357d9fe29ab4ec79f2d78904e7877e1ec779d37eea"
Oct 30 21:30:20.108966 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:20.107365    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx_odh-ai-gateway-infra(cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1"
Oct 30 21:30:20.108966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.107418    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css" event={"ID":"7a4e9f94-c578-43a7-9f01-937d5cc8c679","Type":"ContainerDied","Data":"b6a8e2d632a54e19a3338f2f47f810610b78b922db01bab34f326b25004e842c"}
Oct 30 21:30:20.248123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.248100    4810 scope.go:122] "RemoveContainer" containerID="8693b32576c63472238c146696565272b4d53048a810b99060cde6eb24908fa4"
Oct 30 21:30:20.270025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.270000    4810 scope.go:122] "RemoveContainer" containerID="f031e3e7e46c6b9da248974156a79b26b908bd1799dbba943c2975dc90b038ac"
Oct 30 21:30:20.305923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.302187    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"]
Oct 30 21:30:20.309296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.309260    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1e0f5-29889930-v2css"]
Oct 30 21:30:20.456910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.456372    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"
Oct 30 21:30:20.605132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.604096    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500bb411-3bf5-4e1a-bc15-36d332b0af13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/500bb411-3bf5-4e1a-bc15-36d332b0af13-maas-api-service-ca\") pod \"500bb411-3bf5-4e1a-bc15-36d332b0af13\" (UID: \"500bb411-3bf5-4e1a-bc15-36d332b0af13\") "
Oct 30 21:30:20.605132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.604172    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500bb411-3bf5-4e1a-bc15-36d332b0af13-kube-api-access-qp27b\" (UniqueName: \"kubernetes.io/projected/500bb411-3bf5-4e1a-bc15-36d332b0af13-kube-api-access-qp27b\") pod \"500bb411-3bf5-4e1a-bc15-36d332b0af13\" (UID: \"500bb411-3bf5-4e1a-bc15-36d332b0af13\") "
Oct 30 21:30:20.605318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.605290    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500bb411-3bf5-4e1a-bc15-36d332b0af13-maas-api-service-ca" pod "500bb411-3bf5-4e1a-bc15-36d332b0af13" (UID: "500bb411-3bf5-4e1a-bc15-36d332b0af13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:20.654936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.654872    4810 scope.go:122] "RemoveContainer" containerID="1b750e3b1321355eeb2b10432a5967fc1d716832017dbebcf2d8a2f2b069264a"
Oct 30 21:30:20.655085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.654973    4810 scope.go:122] "RemoveContainer" containerID="f2d8f91119244a3918b966d3627d8f9795d5e6114894461006c8594beb7ed87f"
Oct 30 21:30:20.672382 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.672324    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500bb411-3bf5-4e1a-bc15-36d332b0af13-kube-api-access-qp27b" pod "500bb411-3bf5-4e1a-bc15-36d332b0af13" (UID: "500bb411-3bf5-4e1a-bc15-36d332b0af13"). InnerVolumeSpecName "kube-api-access-qp27b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:20.688556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.688527    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"
Oct 30 21:30:20.697069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.697044    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"
Oct 30 21:30:20.724893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.724846    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/500bb411-3bf5-4e1a-bc15-36d332b0af13-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:20.725021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.724896    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp27b\" (UniqueName: \"kubernetes.io/projected/500bb411-3bf5-4e1a-bc15-36d332b0af13-kube-api-access-qp27b\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:20.826192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.826156    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052961e9-c81b-4273-94b9-4704b21dba4f-kube-api-access-mcc2p\" (UniqueName: \"kubernetes.io/projected/052961e9-c81b-4273-94b9-4704b21dba4f-kube-api-access-mcc2p\") pod \"052961e9-c81b-4273-94b9-4704b21dba4f\" (UID: \"052961e9-c81b-4273-94b9-4704b21dba4f\") "
Oct 30 21:30:20.826345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.826248    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052961e9-c81b-4273-94b9-4704b21dba4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052961e9-c81b-4273-94b9-4704b21dba4f-maas-api-service-ca\") pod \"052961e9-c81b-4273-94b9-4704b21dba4f\" (UID: \"052961e9-c81b-4273-94b9-4704b21dba4f\") "
Oct 30 21:30:20.826441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.826394    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a5aa95-faba-4af6-a80e-137745490e73-kube-api-access-xwxk5\" (UniqueName: \"kubernetes.io/projected/35a5aa95-faba-4af6-a80e-137745490e73-kube-api-access-xwxk5\") pod \"35a5aa95-faba-4af6-a80e-137745490e73\" (UID: \"35a5aa95-faba-4af6-a80e-137745490e73\") "
Oct 30 21:30:20.826652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.826628    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a5aa95-faba-4af6-a80e-137745490e73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a5aa95-faba-4af6-a80e-137745490e73-maas-api-service-ca\") pod \"35a5aa95-faba-4af6-a80e-137745490e73\" (UID: \"35a5aa95-faba-4af6-a80e-137745490e73\") "
Oct 30 21:30:20.826723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.826673    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052961e9-c81b-4273-94b9-4704b21dba4f-maas-api-service-ca" pod "052961e9-c81b-4273-94b9-4704b21dba4f" (UID: "052961e9-c81b-4273-94b9-4704b21dba4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:20.827148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.827122    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a5aa95-faba-4af6-a80e-137745490e73-maas-api-service-ca" pod "35a5aa95-faba-4af6-a80e-137745490e73" (UID: "35a5aa95-faba-4af6-a80e-137745490e73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:20.827473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.827446    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a5aa95-faba-4af6-a80e-137745490e73-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:20.827563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.827475    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052961e9-c81b-4273-94b9-4704b21dba4f-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:20.829284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.829163    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052961e9-c81b-4273-94b9-4704b21dba4f-kube-api-access-mcc2p" pod "052961e9-c81b-4273-94b9-4704b21dba4f" (UID: "052961e9-c81b-4273-94b9-4704b21dba4f"). InnerVolumeSpecName "kube-api-access-mcc2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:20.829610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.829473    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a5aa95-faba-4af6-a80e-137745490e73-kube-api-access-xwxk5" pod "35a5aa95-faba-4af6-a80e-137745490e73" (UID: "35a5aa95-faba-4af6-a80e-137745490e73"). InnerVolumeSpecName "kube-api-access-xwxk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:20.929195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.929110    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcc2p\" (UniqueName: \"kubernetes.io/projected/052961e9-c81b-4273-94b9-4704b21dba4f-kube-api-access-mcc2p\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:20.929195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:20.929146    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwxk5\" (UniqueName: \"kubernetes.io/projected/35a5aa95-faba-4af6-a80e-137745490e73-kube-api-access-xwxk5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:21.118841 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.118802    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"]
Oct 30 21:30:21.125378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.125350    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"
Oct 30 21:30:21.125566 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.125427    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl" event={"ID":"35a5aa95-faba-4af6-a80e-137745490e73","Type":"ContainerDied","Data":"d609d91381f16030ad875ee67f33b19542aad08787b1dc024a1a82c18c8b90a2"}
Oct 30 21:30:21.125566 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.125476    4810 scope.go:122] "RemoveContainer" containerID="981c95d94f41d7ac88f09c6050ff7aed393ffef3a572bf06c91c505446362c6f"
Oct 30 21:30:21.136174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.136148    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"
Oct 30 21:30:21.136308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.136178    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv" event={"ID":"052961e9-c81b-4273-94b9-4704b21dba4f","Type":"ContainerDied","Data":"8b20476e6a3c3ff11d4ca84026670197f999410e234068fde86e202c5d171a39"}
Oct 30 21:30:21.138158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.138123    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl" event={"ID":"500bb411-3bf5-4e1a-bc15-36d332b0af13","Type":"ContainerDied","Data":"1c925ff586a0bc807d2034b386370548941b395797adbb0dcce5ffa88d51635c"}
Oct 30 21:30:21.138290 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.138165    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"
Oct 30 21:30:21.140109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.140086    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" event={"ID":"112b0e88-a621-4dc9-bf1f-1099de0cfe5d","Type":"ContainerStarted","Data":"5adc910c5826cc717ea29972a8fcdb305fa9234fb1ea5cc164f0ed52e8395890"}
Oct 30 21:30:21.142182 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.142160    4810 generic.go:356] "Generic (PLEG): container finished" podID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerID="7c083e82ea583dcfb529a5890d827a189d924e2ccab282e432e7a457052e82c6" exitCode=60
Oct 30 21:30:21.142304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.142238    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62" event={"ID":"49e54b96-7b3a-421c-9ca0-312047e92c64","Type":"ContainerDied","Data":"7c083e82ea583dcfb529a5890d827a189d924e2ccab282e432e7a457052e82c6"}
Oct 30 21:30:21.146507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.146477    4810 generic.go:356] "Generic (PLEG): container finished" podID="daad11ef-c385-4cf8-975b-e63912445add" containerID="d8474e468c30f9565ff52cd0602a6edbca80f420d078054d3972b2c60f53a4e2" exitCode=60
Oct 30 21:30:21.146634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.146577    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h" event={"ID":"daad11ef-c385-4cf8-975b-e63912445add","Type":"ContainerDied","Data":"d8474e468c30f9565ff52cd0602a6edbca80f420d078054d3972b2c60f53a4e2"}
Oct 30 21:30:21.147245 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.147225    4810 scope.go:122] "RemoveContainer" containerID="d8474e468c30f9565ff52cd0602a6edbca80f420d078054d3972b2c60f53a4e2"
Oct 30 21:30:21.147645 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:21.147552    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h_odh-ai-gateway-infra(daad11ef-c385-4cf8-975b-e63912445add)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h" podUID="daad11ef-c385-4cf8-975b-e63912445add"
Oct 30 21:30:21.149100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.149067    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" event={"ID":"1ac65f6a-7beb-4c50-a0ba-546af2072612","Type":"ContainerStarted","Data":"6547ea12706bd5459feec1cdc657b88eeaa56a563e6454c3da5901198f51993e"}
Oct 30 21:30:21.151271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.151245    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" event={"ID":"8fd73835-f5e2-47b8-8403-3aa7f03492eb","Type":"ContainerStarted","Data":"df9124f60ce91becdf25cc3cf31495454434158ed95a5eb571568a5f1596d3bc"}
Oct 30 21:30:21.153157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.153138    4810 generic.go:356] "Generic (PLEG): container finished" podID="aca678bf-105c-472c-90e8-3226bdde7e25" containerID="5836759ef1ffa29122ca75e0bb3cb60646114853e2736381f0efaa20d4f89fe2" exitCode=60
Oct 30 21:30:21.153262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.153203    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25" event={"ID":"aca678bf-105c-472c-90e8-3226bdde7e25","Type":"ContainerDied","Data":"5836759ef1ffa29122ca75e0bb3cb60646114853e2736381f0efaa20d4f89fe2"}
Oct 30 21:30:21.153517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.153488    4810 scope.go:122] "RemoveContainer" containerID="5836759ef1ffa29122ca75e0bb3cb60646114853e2736381f0efaa20d4f89fe2"
Oct 30 21:30:21.153740 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:21.153720    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25_odh-ai-gateway-infra(aca678bf-105c-472c-90e8-3226bdde7e25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25" podUID="aca678bf-105c-472c-90e8-3226bdde7e25"
Oct 30 21:30:21.155139 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.155122    4810 generic.go:356] "Generic (PLEG): container finished" podID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerID="cfad3f4a43d9fc505c7eb9d0c85044a5ef1bcddc30c303584bf5ebb3190d168b" exitCode=60
Oct 30 21:30:21.155224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.155197    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" event={"ID":"aaf881a3-86a2-494d-b8e5-3a789801bd41","Type":"ContainerDied","Data":"cfad3f4a43d9fc505c7eb9d0c85044a5ef1bcddc30c303584bf5ebb3190d168b"}
Oct 30 21:30:21.155487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.155473    4810 scope.go:122] "RemoveContainer" containerID="cfad3f4a43d9fc505c7eb9d0c85044a5ef1bcddc30c303584bf5ebb3190d168b"
Oct 30 21:30:21.155655 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:21.155634    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp_odh-ai-gateway-infra(aaf881a3-86a2-494d-b8e5-3a789801bd41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41"
Oct 30 21:30:21.157039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.157021    4810 generic.go:356] "Generic (PLEG): container finished" podID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerID="1053669f93d4ce15b5e66a357d9fe29ab4ec79f2d78904e7877e1ec779d37eea" exitCode=60
Oct 30 21:30:21.157125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.157074    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" event={"ID":"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1","Type":"ContainerDied","Data":"1053669f93d4ce15b5e66a357d9fe29ab4ec79f2d78904e7877e1ec779d37eea"}
Oct 30 21:30:21.157335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.157321    4810 scope.go:122] "RemoveContainer" containerID="1053669f93d4ce15b5e66a357d9fe29ab4ec79f2d78904e7877e1ec779d37eea"
Oct 30 21:30:21.157541 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:21.157524    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx_odh-ai-gateway-infra(cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1"
Oct 30 21:30:21.159808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.159790    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p" event={"ID":"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380","Type":"ContainerStarted","Data":"01359325c672f6df7ce9018ee649703e90afcfaa9ee311ec6859f90850d6eed9"}
Oct 30 21:30:21.189923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.189899    4810 scope.go:122] "RemoveContainer" containerID="8eb477b2797a7786057df2e533e54b82c151dad2ed1d5227965d2ec6cdb00b25"
Oct 30 21:30:21.196056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.196033    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"]
Oct 30 21:30:21.209536 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.209507    4810 scope.go:122] "RemoveContainer" containerID="e00432a425cbdbd29259983d9aec53c4a78f79fb309112d18c113c55ec779a97"
Oct 30 21:30:21.226957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.226927    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"]
Oct 30 21:30:21.228321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.228299    4810 scope.go:122] "RemoveContainer" containerID="b04032598659c1205037a51e328afbedb6515ef9d1c12c9ba89ca157b499a470"
Oct 30 21:30:21.239818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.239788    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"]
Oct 30 21:30:21.245146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.245127    4810 scope.go:122] "RemoveContainer" containerID="6204d4ef75cf584cf72ffe2164af6844c1d67cc6014f693f42b0782ae4c89c12"
Oct 30 21:30:21.260895 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.258954    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5506f4-29889930-8hmbl"]
Oct 30 21:30:21.265100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.264027    4810 scope.go:122] "RemoveContainer" containerID="1ad66ef0024e4b3dbac652719a710edaa1b80ec51afc1b597f2d8daec197b125"
Oct 30 21:30:21.307348 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.307313    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"]
Oct 30 21:30:21.335926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.319922    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"]
Oct 30 21:30:21.335926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.328280    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-321ccd-29889930-xh7rl"]
Oct 30 21:30:21.385005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.384896    4810 scope.go:122] "RemoveContainer" containerID="453678314e52b76c6ebc8ca8a359487755a4433532c543eb9b30b62be3a55ff3"
Oct 30 21:30:21.387387 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.386735    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"]
Oct 30 21:30:21.393600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.393558    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-szvjv"]
Oct 30 21:30:21.394874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.394809    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p" podStartSLOduration=21.394790977 podStartE2EDuration="21.394790977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:21.366588029 +0000 UTC m=+3187.858404575" watchObservedRunningTime="2026-10-30 21:30:21.394790977 +0000 UTC m=+3187.886607523"
Oct 30 21:30:21.530820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.530797    4810 scope.go:122] "RemoveContainer" containerID="5cd627d7a9d5639088d95625fb2ee1e101a21a9477dd379dc92a7dfec890aea9"
Oct 30 21:30:21.543732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.543708    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"
Oct 30 21:30:21.571610 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.571581    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e54b96-7b3a-421c-9ca0-312047e92c64-kube-api-access-zj6fn\" (UniqueName: \"kubernetes.io/projected/49e54b96-7b3a-421c-9ca0-312047e92c64-kube-api-access-zj6fn\") pod \"49e54b96-7b3a-421c-9ca0-312047e92c64\" (UID: \"49e54b96-7b3a-421c-9ca0-312047e92c64\") "
Oct 30 21:30:21.572234 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.571843    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e54b96-7b3a-421c-9ca0-312047e92c64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e54b96-7b3a-421c-9ca0-312047e92c64-maas-api-service-ca\") pod \"49e54b96-7b3a-421c-9ca0-312047e92c64\" (UID: \"49e54b96-7b3a-421c-9ca0-312047e92c64\") "
Oct 30 21:30:21.572582 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.572202    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e54b96-7b3a-421c-9ca0-312047e92c64-maas-api-service-ca" pod "49e54b96-7b3a-421c-9ca0-312047e92c64" (UID: "49e54b96-7b3a-421c-9ca0-312047e92c64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:21.573356 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.573333    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e54b96-7b3a-421c-9ca0-312047e92c64-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:21.574635 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.574607    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e54b96-7b3a-421c-9ca0-312047e92c64-kube-api-access-zj6fn" pod "49e54b96-7b3a-421c-9ca0-312047e92c64" (UID: "49e54b96-7b3a-421c-9ca0-312047e92c64"). InnerVolumeSpecName "kube-api-access-zj6fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:21.693207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.693111    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj6fn\" (UniqueName: \"kubernetes.io/projected/49e54b96-7b3a-421c-9ca0-312047e92c64-kube-api-access-zj6fn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:21.697538 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.697506    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052961e9-c81b-4273-94b9-4704b21dba4f" path="/var/lib/kubelet/pods/052961e9-c81b-4273-94b9-4704b21dba4f/volumes"
Oct 30 21:30:21.698162 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.698137    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a5aa95-faba-4af6-a80e-137745490e73" path="/var/lib/kubelet/pods/35a5aa95-faba-4af6-a80e-137745490e73/volumes"
Oct 30 21:30:21.698654 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.698636    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13" path="/var/lib/kubelet/pods/500bb411-3bf5-4e1a-bc15-36d332b0af13/volumes"
Oct 30 21:30:21.699844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:21.699645    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" path="/var/lib/kubelet/pods/7a4e9f94-c578-43a7-9f01-937d5cc8c679/volumes"
Oct 30 21:30:22.170682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.170652    4810 generic.go:356] "Generic (PLEG): container finished" podID="64c97f25-107d-43c7-ad29-52fac553bedc" containerID="bcc1bffd9da9784f5a2e34170f7000065f71ecac7285babce312f16290987b63" exitCode=60
Oct 30 21:30:22.170886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.170725    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj" event={"ID":"64c97f25-107d-43c7-ad29-52fac553bedc","Type":"ContainerDied","Data":"bcc1bffd9da9784f5a2e34170f7000065f71ecac7285babce312f16290987b63"}
Oct 30 21:30:22.170886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.170794    4810 scope.go:122] "RemoveContainer" containerID="1b750e3b1321355eeb2b10432a5967fc1d716832017dbebcf2d8a2f2b069264a"
Oct 30 21:30:22.171364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.171337    4810 scope.go:122] "RemoveContainer" containerID="bcc1bffd9da9784f5a2e34170f7000065f71ecac7285babce312f16290987b63"
Oct 30 21:30:22.171781 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:22.171744    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj_odh-ai-gateway-infra(64c97f25-107d-43c7-ad29-52fac553bedc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj" podUID="64c97f25-107d-43c7-ad29-52fac553bedc"
Oct 30 21:30:22.177062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.177038    4810 generic.go:356] "Generic (PLEG): container finished" podID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerID="5adc910c5826cc717ea29972a8fcdb305fa9234fb1ea5cc164f0ed52e8395890" exitCode=60
Oct 30 21:30:22.177186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.177070    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" event={"ID":"112b0e88-a621-4dc9-bf1f-1099de0cfe5d","Type":"ContainerDied","Data":"5adc910c5826cc717ea29972a8fcdb305fa9234fb1ea5cc164f0ed52e8395890"}
Oct 30 21:30:22.177561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.177539    4810 scope.go:122] "RemoveContainer" containerID="5adc910c5826cc717ea29972a8fcdb305fa9234fb1ea5cc164f0ed52e8395890"
Oct 30 21:30:22.178012 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:22.177985    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx_odh-ai-gateway-infra(112b0e88-a621-4dc9-bf1f-1099de0cfe5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d"
Oct 30 21:30:22.179042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.178997    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"
Oct 30 21:30:22.179042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.179021    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62" event={"ID":"49e54b96-7b3a-421c-9ca0-312047e92c64","Type":"ContainerDied","Data":"77226071ca626eea89d3b35184bdb5a4721f7f4a93841c983ab9eb0f48368c92"}
Oct 30 21:30:22.185183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.185158    4810 generic.go:356] "Generic (PLEG): container finished" podID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerID="6547ea12706bd5459feec1cdc657b88eeaa56a563e6454c3da5901198f51993e" exitCode=60
Oct 30 21:30:22.185286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.185262    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" event={"ID":"1ac65f6a-7beb-4c50-a0ba-546af2072612","Type":"ContainerDied","Data":"6547ea12706bd5459feec1cdc657b88eeaa56a563e6454c3da5901198f51993e"}
Oct 30 21:30:22.185673 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.185655    4810 scope.go:122] "RemoveContainer" containerID="6547ea12706bd5459feec1cdc657b88eeaa56a563e6454c3da5901198f51993e"
Oct 30 21:30:22.186053 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:22.186024    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r_odh-ai-gateway-infra(1ac65f6a-7beb-4c50-a0ba-546af2072612)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612"
Oct 30 21:30:22.189677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.189580    4810 scope.go:122] "RemoveContainer" containerID="a3195a969e93ca8b5ea51d015661a564c83c8bfb30a5d37d6b4fcdc4eca7fcb1"
Oct 30 21:30:22.202228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.201438    4810 generic.go:356] "Generic (PLEG): container finished" podID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerID="e3c5a3d726e301bd6eb7c29f53461427a863509e7ff0a336a7f74c36267208e3" exitCode=60
Oct 30 21:30:22.202228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.201583    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c" event={"ID":"40bb8f64-ee03-4754-89ed-5c69010f4eba","Type":"ContainerDied","Data":"e3c5a3d726e301bd6eb7c29f53461427a863509e7ff0a336a7f74c36267208e3"}
Oct 30 21:30:22.202909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.202885    4810 scope.go:122] "RemoveContainer" containerID="e3c5a3d726e301bd6eb7c29f53461427a863509e7ff0a336a7f74c36267208e3"
Oct 30 21:30:22.218711 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:22.218669    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c_odh-ai-gateway-infra(40bb8f64-ee03-4754-89ed-5c69010f4eba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba"
Oct 30 21:30:22.220247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.220215    4810 generic.go:356] "Generic (PLEG): container finished" podID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerID="df9124f60ce91becdf25cc3cf31495454434158ed95a5eb571568a5f1596d3bc" exitCode=60
Oct 30 21:30:22.220395 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.220327    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" event={"ID":"8fd73835-f5e2-47b8-8403-3aa7f03492eb","Type":"ContainerDied","Data":"df9124f60ce91becdf25cc3cf31495454434158ed95a5eb571568a5f1596d3bc"}
Oct 30 21:30:22.220847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.220792    4810 scope.go:122] "RemoveContainer" containerID="df9124f60ce91becdf25cc3cf31495454434158ed95a5eb571568a5f1596d3bc"
Oct 30 21:30:22.221292 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:22.221262    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7_odh-ai-gateway-infra(8fd73835-f5e2-47b8-8403-3aa7f03492eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb"
Oct 30 21:30:22.221292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.221280    4810 scope.go:122] "RemoveContainer" containerID="7c083e82ea583dcfb529a5890d827a189d924e2ccab282e432e7a457052e82c6"
Oct 30 21:30:22.237563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.237452    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"]
Oct 30 21:30:22.245633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.245560    4810 generic.go:356] "Generic (PLEG): container finished" podID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerID="01359325c672f6df7ce9018ee649703e90afcfaa9ee311ec6859f90850d6eed9" exitCode=60
Oct 30 21:30:22.245751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.245696    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p" event={"ID":"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380","Type":"ContainerDied","Data":"01359325c672f6df7ce9018ee649703e90afcfaa9ee311ec6859f90850d6eed9"}
Oct 30 21:30:22.246623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.246595    4810 scope.go:122] "RemoveContainer" containerID="01359325c672f6df7ce9018ee649703e90afcfaa9ee311ec6859f90850d6eed9"
Oct 30 21:30:22.246976 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:22.246925    4810 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p_odh-ai-gateway-infra(cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380"
Oct 30 21:30:22.247247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.247212    4810 scope.go:122] "RemoveContainer" containerID="37211fa0765cc80d3e1a9304545a1273a88902b91390dc994dac058706cddf5c"
Oct 30 21:30:22.260718 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.260674    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"]
Oct 30 21:30:22.268112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.268090    4810 scope.go:122] "RemoveContainer" containerID="f2d8f91119244a3918b966d3627d8f9795d5e6114894461006c8594beb7ed87f"
Oct 30 21:30:22.277020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.276592    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-bc102e-29889930-f5w62"]
Oct 30 21:30:22.321731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.296027    4810 scope.go:122] "RemoveContainer" containerID="f5cf98748bad58007a2c4a569e3b1071684d405b09a95b58218ce5b67b3f3046"
Oct 30 21:30:22.353298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.353266    4810 scope.go:122] "RemoveContainer" containerID="e843271db4acd069e603933378531896f404bd6dfef528a860f4602f88959b0f"
Oct 30 21:30:22.365138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.365107    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"]
Oct 30 21:30:22.407393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.407361    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"]
Oct 30 21:30:22.449751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.449720    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"]
Oct 30 21:30:22.472825 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.472246    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"]
Oct 30 21:30:22.474266 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.474145    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"
Oct 30 21:30:22.539405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.539374    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daad11ef-c385-4cf8-975b-e63912445add-kube-api-access-89bsd\" (UniqueName: \"kubernetes.io/projected/daad11ef-c385-4cf8-975b-e63912445add-kube-api-access-89bsd\") pod \"daad11ef-c385-4cf8-975b-e63912445add\" (UID: \"daad11ef-c385-4cf8-975b-e63912445add\") "
Oct 30 21:30:22.541767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.541749    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daad11ef-c385-4cf8-975b-e63912445add-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad11ef-c385-4cf8-975b-e63912445add-maas-api-service-ca\") pod \"daad11ef-c385-4cf8-975b-e63912445add\" (UID: \"daad11ef-c385-4cf8-975b-e63912445add\") "
Oct 30 21:30:22.542318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.542291    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daad11ef-c385-4cf8-975b-e63912445add-maas-api-service-ca" pod "daad11ef-c385-4cf8-975b-e63912445add" (UID: "daad11ef-c385-4cf8-975b-e63912445add"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:22.542582 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.542566    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad11ef-c385-4cf8-975b-e63912445add-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.546660 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.546622    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daad11ef-c385-4cf8-975b-e63912445add-kube-api-access-89bsd" pod "daad11ef-c385-4cf8-975b-e63912445add" (UID: "daad11ef-c385-4cf8-975b-e63912445add"). InnerVolumeSpecName "kube-api-access-89bsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.648547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.648517    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89bsd\" (UniqueName: \"kubernetes.io/projected/daad11ef-c385-4cf8-975b-e63912445add-kube-api-access-89bsd\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.652921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.652848    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"
Oct 30 21:30:22.655196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.655176    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"
Oct 30 21:30:22.659612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.659591    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"
Oct 30 21:30:22.750513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.750478    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf881a3-86a2-494d-b8e5-3a789801bd41-kube-api-access-4qf98\" (UniqueName: \"kubernetes.io/projected/aaf881a3-86a2-494d-b8e5-3a789801bd41-kube-api-access-4qf98\") pod \"aaf881a3-86a2-494d-b8e5-3a789801bd41\" (UID: \"aaf881a3-86a2-494d-b8e5-3a789801bd41\") "
Oct 30 21:30:22.752082 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.752034    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-kube-api-access-gfjdn\" (UniqueName: \"kubernetes.io/projected/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-kube-api-access-gfjdn\") pod \"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1\" (UID: \"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1\") "
Oct 30 21:30:22.752281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.752256    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca678bf-105c-472c-90e8-3226bdde7e25-kube-api-access-lltx4\" (UniqueName: \"kubernetes.io/projected/aca678bf-105c-472c-90e8-3226bdde7e25-kube-api-access-lltx4\") pod \"aca678bf-105c-472c-90e8-3226bdde7e25\" (UID: \"aca678bf-105c-472c-90e8-3226bdde7e25\") "
Oct 30 21:30:22.752368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.752284    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf881a3-86a2-494d-b8e5-3a789801bd41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf881a3-86a2-494d-b8e5-3a789801bd41-maas-api-service-ca\") pod \"aaf881a3-86a2-494d-b8e5-3a789801bd41\" (UID: \"aaf881a3-86a2-494d-b8e5-3a789801bd41\") "
Oct 30 21:30:22.752368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.752308    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca678bf-105c-472c-90e8-3226bdde7e25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca678bf-105c-472c-90e8-3226bdde7e25-maas-api-service-ca\") pod \"aca678bf-105c-472c-90e8-3226bdde7e25\" (UID: \"aca678bf-105c-472c-90e8-3226bdde7e25\") "
Oct 30 21:30:22.752537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.752511    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-maas-api-service-ca\") pod \"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1\" (UID: \"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1\") "
Oct 30 21:30:22.752717 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.752683    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf881a3-86a2-494d-b8e5-3a789801bd41-maas-api-service-ca" pod "aaf881a3-86a2-494d-b8e5-3a789801bd41" (UID: "aaf881a3-86a2-494d-b8e5-3a789801bd41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:22.752717 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.752693    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca678bf-105c-472c-90e8-3226bdde7e25-maas-api-service-ca" pod "aca678bf-105c-472c-90e8-3226bdde7e25" (UID: "aca678bf-105c-472c-90e8-3226bdde7e25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:22.753696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.753117    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-maas-api-service-ca" pod "cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" (UID: "cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:22.753793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.753699    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.753793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.753724    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf881a3-86a2-494d-b8e5-3a789801bd41-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.753793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.753739    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca678bf-105c-472c-90e8-3226bdde7e25-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.754793 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.754672    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf881a3-86a2-494d-b8e5-3a789801bd41-kube-api-access-4qf98" pod "aaf881a3-86a2-494d-b8e5-3a789801bd41" (UID: "aaf881a3-86a2-494d-b8e5-3a789801bd41"). InnerVolumeSpecName "kube-api-access-4qf98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.761328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.761300    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca678bf-105c-472c-90e8-3226bdde7e25-kube-api-access-lltx4" pod "aca678bf-105c-472c-90e8-3226bdde7e25" (UID: "aca678bf-105c-472c-90e8-3226bdde7e25"). InnerVolumeSpecName "kube-api-access-lltx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.761734 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.761712    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-kube-api-access-gfjdn" pod "cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" (UID: "cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1"). InnerVolumeSpecName "kube-api-access-gfjdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.855277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.855199    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lltx4\" (UniqueName: \"kubernetes.io/projected/aca678bf-105c-472c-90e8-3226bdde7e25-kube-api-access-lltx4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.855277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.855229    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qf98\" (UniqueName: \"kubernetes.io/projected/aaf881a3-86a2-494d-b8e5-3a789801bd41-kube-api-access-4qf98\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.855277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:22.855239    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfjdn\" (UniqueName: \"kubernetes.io/projected/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1-kube-api-access-gfjdn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:23.200987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.200955    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"]
Oct 30 21:30:23.262490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.262464    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"
Oct 30 21:30:23.262746 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.262474    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h" event={"ID":"daad11ef-c385-4cf8-975b-e63912445add","Type":"ContainerDied","Data":"2102c8242d06999057a3f0baa84b6771408d20f70483274c0de027f54908b0af"}
Oct 30 21:30:23.262836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.262791    4810 scope.go:122] "RemoveContainer" containerID="d8474e468c30f9565ff52cd0602a6edbca80f420d078054d3972b2c60f53a4e2"
Oct 30 21:30:23.274754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.274728    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25" event={"ID":"aca678bf-105c-472c-90e8-3226bdde7e25","Type":"ContainerDied","Data":"cfec8f4e3323112992112e8c9b4ac908a3f95ad19a005ecfdcd4a428ad8730c4"}
Oct 30 21:30:23.275666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.275646    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"
Oct 30 21:30:23.280755 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.280735    4810 scope.go:122] "RemoveContainer" containerID="5836759ef1ffa29122ca75e0bb3cb60646114853e2736381f0efaa20d4f89fe2"
Oct 30 21:30:23.307927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.307312    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp" event={"ID":"aaf881a3-86a2-494d-b8e5-3a789801bd41","Type":"ContainerDied","Data":"de3665607596fe36fa7b824f1d7cd8ca616ddb798969d13da4105061738699d4"}
Oct 30 21:30:23.307927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.307417    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"
Oct 30 21:30:23.322138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.316954    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"
Oct 30 21:30:23.322138 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.317957    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx" event={"ID":"cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1","Type":"ContainerDied","Data":"deb7209506b8d426d634f0a9dbc7f93a7180574d8f314f0c5f974170531a8014"}
Oct 30 21:30:23.391710 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.391348    4810 scope.go:122] "RemoveContainer" containerID="cfad3f4a43d9fc505c7eb9d0c85044a5ef1bcddc30c303584bf5ebb3190d168b"
Oct 30 21:30:23.420135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.420091    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"]
Oct 30 21:30:23.463322 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.462095    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"]
Oct 30 21:30:23.463473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.463449    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8fa15b-29889930-blt25"]
Oct 30 21:30:23.478545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.478520    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"]
Oct 30 21:30:23.487648 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.487586    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8e2c3-29889930-kcrcx"]
Oct 30 21:30:23.500672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.500579    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"]
Oct 30 21:30:23.511307 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.511282    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889930-jlw5h"]
Oct 30 21:30:23.521321 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.521299    4810 scope.go:122] "RemoveContainer" containerID="1053669f93d4ce15b5e66a357d9fe29ab4ec79f2d78904e7877e1ec779d37eea"
Oct 30 21:30:23.560730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.559669    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"]
Oct 30 21:30:23.581618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.581583    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f64640-29889930-svsrp"]
Oct 30 21:30:23.709823 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.709788    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64" path="/var/lib/kubelet/pods/49e54b96-7b3a-421c-9ca0-312047e92c64/volumes"
Oct 30 21:30:23.711560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.711534    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41" path="/var/lib/kubelet/pods/aaf881a3-86a2-494d-b8e5-3a789801bd41/volumes"
Oct 30 21:30:23.712605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.712568    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca678bf-105c-472c-90e8-3226bdde7e25" path="/var/lib/kubelet/pods/aca678bf-105c-472c-90e8-3226bdde7e25/volumes"
Oct 30 21:30:23.714985 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.714913    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" path="/var/lib/kubelet/pods/cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1/volumes"
Oct 30 21:30:23.715695 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.715675    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daad11ef-c385-4cf8-975b-e63912445add" path="/var/lib/kubelet/pods/daad11ef-c385-4cf8-975b-e63912445add/volumes"
Oct 30 21:30:23.726275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.726249    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"
Oct 30 21:30:23.762907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.761303    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"
Oct 30 21:30:23.825476 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.825450    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd73835-f5e2-47b8-8403-3aa7f03492eb-kube-api-access-jvnjg\" (UniqueName: \"kubernetes.io/projected/8fd73835-f5e2-47b8-8403-3aa7f03492eb-kube-api-access-jvnjg\") pod \"8fd73835-f5e2-47b8-8403-3aa7f03492eb\" (UID: \"8fd73835-f5e2-47b8-8403-3aa7f03492eb\") "
Oct 30 21:30:23.825850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.825830    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd73835-f5e2-47b8-8403-3aa7f03492eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd73835-f5e2-47b8-8403-3aa7f03492eb-maas-api-service-ca\") pod \"8fd73835-f5e2-47b8-8403-3aa7f03492eb\" (UID: \"8fd73835-f5e2-47b8-8403-3aa7f03492eb\") "
Oct 30 21:30:23.826313 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.826279    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd73835-f5e2-47b8-8403-3aa7f03492eb-maas-api-service-ca" pod "8fd73835-f5e2-47b8-8403-3aa7f03492eb" (UID: "8fd73835-f5e2-47b8-8403-3aa7f03492eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:23.826637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.826616    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd73835-f5e2-47b8-8403-3aa7f03492eb-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:23.828638 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.828615    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd73835-f5e2-47b8-8403-3aa7f03492eb-kube-api-access-jvnjg" pod "8fd73835-f5e2-47b8-8403-3aa7f03492eb" (UID: "8fd73835-f5e2-47b8-8403-3aa7f03492eb"). InnerVolumeSpecName "kube-api-access-jvnjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:23.885490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.885461    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"
Oct 30 21:30:23.887575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.887553    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"
Oct 30 21:30:23.889786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.889765    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"
Oct 30 21:30:23.895112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.895090    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"
Oct 30 21:30:23.927619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.927590    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-maas-api-service-ca\") pod \"112b0e88-a621-4dc9-bf1f-1099de0cfe5d\" (UID: \"112b0e88-a621-4dc9-bf1f-1099de0cfe5d\") "
Oct 30 21:30:23.927933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.927910    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-kube-api-access-qv928\" (UniqueName: \"kubernetes.io/projected/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-kube-api-access-qv928\") pod \"112b0e88-a621-4dc9-bf1f-1099de0cfe5d\" (UID: \"112b0e88-a621-4dc9-bf1f-1099de0cfe5d\") "
Oct 30 21:30:23.928687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.928665    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvnjg\" (UniqueName: \"kubernetes.io/projected/8fd73835-f5e2-47b8-8403-3aa7f03492eb-kube-api-access-jvnjg\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:23.930727 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.930700    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-kube-api-access-qv928" pod "112b0e88-a621-4dc9-bf1f-1099de0cfe5d" (UID: "112b0e88-a621-4dc9-bf1f-1099de0cfe5d"). InnerVolumeSpecName "kube-api-access-qv928". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:23.934950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:23.934916    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-maas-api-service-ca" pod "112b0e88-a621-4dc9-bf1f-1099de0cfe5d" (UID: "112b0e88-a621-4dc9-bf1f-1099de0cfe5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.029897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.029770    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac65f6a-7beb-4c50-a0ba-546af2072612-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac65f6a-7beb-4c50-a0ba-546af2072612-maas-api-service-ca\") pod \"1ac65f6a-7beb-4c50-a0ba-546af2072612\" (UID: \"1ac65f6a-7beb-4c50-a0ba-546af2072612\") "
Oct 30 21:30:24.030065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.029907    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac65f6a-7beb-4c50-a0ba-546af2072612-kube-api-access-swmff\" (UniqueName: \"kubernetes.io/projected/1ac65f6a-7beb-4c50-a0ba-546af2072612-kube-api-access-swmff\") pod \"1ac65f6a-7beb-4c50-a0ba-546af2072612\" (UID: \"1ac65f6a-7beb-4c50-a0ba-546af2072612\") "
Oct 30 21:30:24.030065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.029949    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c97f25-107d-43c7-ad29-52fac553bedc-kube-api-access-fnwxf\" (UniqueName: \"kubernetes.io/projected/64c97f25-107d-43c7-ad29-52fac553bedc-kube-api-access-fnwxf\") pod \"64c97f25-107d-43c7-ad29-52fac553bedc\" (UID: \"64c97f25-107d-43c7-ad29-52fac553bedc\") "
Oct 30 21:30:24.030065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.030008    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-maas-api-service-ca\") pod \"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380\" (UID: \"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380\") "
Oct 30 21:30:24.030065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.030048    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40bb8f64-ee03-4754-89ed-5c69010f4eba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb8f64-ee03-4754-89ed-5c69010f4eba-maas-api-service-ca\") pod \"40bb8f64-ee03-4754-89ed-5c69010f4eba\" (UID: \"40bb8f64-ee03-4754-89ed-5c69010f4eba\") "
Oct 30 21:30:24.030306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.030091    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bb8f64-ee03-4754-89ed-5c69010f4eba-kube-api-access-ggjcq\" (UniqueName: \"kubernetes.io/projected/40bb8f64-ee03-4754-89ed-5c69010f4eba-kube-api-access-ggjcq\") pod \"40bb8f64-ee03-4754-89ed-5c69010f4eba\" (UID: \"40bb8f64-ee03-4754-89ed-5c69010f4eba\") "
Oct 30 21:30:24.030306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.030136    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c97f25-107d-43c7-ad29-52fac553bedc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c97f25-107d-43c7-ad29-52fac553bedc-maas-api-service-ca\") pod \"64c97f25-107d-43c7-ad29-52fac553bedc\" (UID: \"64c97f25-107d-43c7-ad29-52fac553bedc\") "
Oct 30 21:30:24.030306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.030187    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac65f6a-7beb-4c50-a0ba-546af2072612-maas-api-service-ca" pod "1ac65f6a-7beb-4c50-a0ba-546af2072612" (UID: "1ac65f6a-7beb-4c50-a0ba-546af2072612"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.030306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.030241    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-kube-api-access-cbmr5\" (UniqueName: \"kubernetes.io/projected/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-kube-api-access-cbmr5\") pod \"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380\" (UID: \"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380\") "
Oct 30 21:30:24.030578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.030556    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-maas-api-service-ca" pod "cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" (UID: "cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.031125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.031098    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bb8f64-ee03-4754-89ed-5c69010f4eba-maas-api-service-ca" pod "40bb8f64-ee03-4754-89ed-5c69010f4eba" (UID: "40bb8f64-ee03-4754-89ed-5c69010f4eba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.031238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.031101    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c97f25-107d-43c7-ad29-52fac553bedc-maas-api-service-ca" pod "64c97f25-107d-43c7-ad29-52fac553bedc" (UID: "64c97f25-107d-43c7-ad29-52fac553bedc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.031304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.031202    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.031304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.031274    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.031304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.031289    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv928\" (UniqueName: \"kubernetes.io/projected/112b0e88-a621-4dc9-bf1f-1099de0cfe5d-kube-api-access-qv928\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.031487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.031306    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac65f6a-7beb-4c50-a0ba-546af2072612-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.032904 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.032840    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bb8f64-ee03-4754-89ed-5c69010f4eba-kube-api-access-ggjcq" pod "40bb8f64-ee03-4754-89ed-5c69010f4eba" (UID: "40bb8f64-ee03-4754-89ed-5c69010f4eba"). InnerVolumeSpecName "kube-api-access-ggjcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.033202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.033166    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-kube-api-access-cbmr5" pod "cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" (UID: "cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380"). InnerVolumeSpecName "kube-api-access-cbmr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.033494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.033469    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac65f6a-7beb-4c50-a0ba-546af2072612-kube-api-access-swmff" pod "1ac65f6a-7beb-4c50-a0ba-546af2072612" (UID: "1ac65f6a-7beb-4c50-a0ba-546af2072612"). InnerVolumeSpecName "kube-api-access-swmff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.033674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.033656    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c97f25-107d-43c7-ad29-52fac553bedc-kube-api-access-fnwxf" pod "64c97f25-107d-43c7-ad29-52fac553bedc" (UID: "64c97f25-107d-43c7-ad29-52fac553bedc"). InnerVolumeSpecName "kube-api-access-fnwxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.132425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.132392    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c97f25-107d-43c7-ad29-52fac553bedc-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.132425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.132421    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbmr5\" (UniqueName: \"kubernetes.io/projected/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380-kube-api-access-cbmr5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.132425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.132432    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmff\" (UniqueName: \"kubernetes.io/projected/1ac65f6a-7beb-4c50-a0ba-546af2072612-kube-api-access-swmff\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.132657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.132442    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnwxf\" (UniqueName: \"kubernetes.io/projected/64c97f25-107d-43c7-ad29-52fac553bedc-kube-api-access-fnwxf\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.132657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.132454    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40bb8f64-ee03-4754-89ed-5c69010f4eba-maas-api-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.132657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.132467    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggjcq\" (UniqueName: \"kubernetes.io/projected/40bb8f64-ee03-4754-89ed-5c69010f4eba-kube-api-access-ggjcq\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.336426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.336329    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"
Oct 30 21:30:24.336426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.336356    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p" event={"ID":"cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380","Type":"ContainerDied","Data":"523f70967f9ae7379807f79cdbfe34e7d4ba79fce3a8b1ff3bb6fad95f0367ad"}
Oct 30 21:30:24.336668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.336434    4810 scope.go:122] "RemoveContainer" containerID="01359325c672f6df7ce9018ee649703e90afcfaa9ee311ec6859f90850d6eed9"
Oct 30 21:30:24.338112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.338093    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"
Oct 30 21:30:24.338239 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.338129    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj" event={"ID":"64c97f25-107d-43c7-ad29-52fac553bedc","Type":"ContainerDied","Data":"41f0a2cf0c310d94527448c990312e2d34afb96f2a531ae58fb01624df43b020"}
Oct 30 21:30:24.344656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.344598    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"
Oct 30 21:30:24.344656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.344634    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx" event={"ID":"112b0e88-a621-4dc9-bf1f-1099de0cfe5d","Type":"ContainerDied","Data":"9888451558d2508720f3ee6c6f2e2aca62ef5772e5f043b084d51a5fed20c7ab"}
Oct 30 21:30:24.348379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.347805    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"
Oct 30 21:30:24.348379 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.347806    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r" event={"ID":"1ac65f6a-7beb-4c50-a0ba-546af2072612","Type":"ContainerDied","Data":"8180e444bb4a5b32db17c8086355ad83c20294fd53112bc251db7d8f61f3e10e"}
Oct 30 21:30:24.351044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.350684    4810 scope.go:122] "RemoveContainer" containerID="bcc1bffd9da9784f5a2e34170f7000065f71ecac7285babce312f16290987b63"
Oct 30 21:30:24.351044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.350702    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"
Oct 30 21:30:24.351044 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.350683    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c" event={"ID":"40bb8f64-ee03-4754-89ed-5c69010f4eba","Type":"ContainerDied","Data":"4bfe2c355bf947d97dbff9d4dfe157bc8f8c6d66cc519e1a1f428b2b3da3b3a4"}
Oct 30 21:30:24.353163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.353042    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7" event={"ID":"8fd73835-f5e2-47b8-8403-3aa7f03492eb","Type":"ContainerDied","Data":"c9a685c901d0c352ba0fae4eba49b5a0c344754ee077c87a1afd432ef6d96d9e"}
Oct 30 21:30:24.353163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.353133    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"
Oct 30 21:30:24.366571 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.366551    4810 scope.go:122] "RemoveContainer" containerID="5adc910c5826cc717ea29972a8fcdb305fa9234fb1ea5cc164f0ed52e8395890"
Oct 30 21:30:24.377454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.377431    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"]
Oct 30 21:30:24.379656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.379637    4810 scope.go:122] "RemoveContainer" containerID="6547ea12706bd5459feec1cdc657b88eeaa56a563e6454c3da5901198f51993e"
Oct 30 21:30:24.385187 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.385168    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e53c37-29889930-p6c2p"]
Oct 30 21:30:24.392335 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.392316    4810 scope.go:122] "RemoveContainer" containerID="e3c5a3d726e301bd6eb7c29f53461427a863509e7ff0a336a7f74c36267208e3"
Oct 30 21:30:24.404901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.404878    4810 scope.go:122] "RemoveContainer" containerID="df9124f60ce91becdf25cc3cf31495454434158ed95a5eb571568a5f1596d3bc"
Oct 30 21:30:24.406944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.406921    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"]
Oct 30 21:30:24.413799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.413778    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0c4052-29889930-gfl8c"]
Oct 30 21:30:24.427612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.427584    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"]
Oct 30 21:30:24.432921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.432899    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d95261-29889930-jn8gj"]
Oct 30 21:30:24.444750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.444725    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"]
Oct 30 21:30:24.448749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.448727    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-317a33-29889930-cbd5r"]
Oct 30 21:30:24.457884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.457846    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"]
Oct 30 21:30:24.466352 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.466325    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99668e-29889930-ln8gx"]
Oct 30 21:30:24.483598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.483570    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"]
Oct 30 21:30:24.489513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:24.489485    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b78e4-29889930-nkjn7"]
Oct 30 21:30:25.162265 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:25.162228    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:30:25.298448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.298420    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:30:25.380885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.380825    4810 generic.go:356] "Generic (PLEG): container finished" podID="9d205dba-ff07-4742-8509-44948ba1af95" containerID="98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e" exitCode=0
Oct 30 21:30:25.381097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.380955    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" event={"ID":"9d205dba-ff07-4742-8509-44948ba1af95","Type":"ContainerDied","Data":"98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e"}
Oct 30 21:30:25.381097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.381006    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp" event={"ID":"9d205dba-ff07-4742-8509-44948ba1af95","Type":"ContainerDied","Data":"bab619f23ab77507b866f8f9bf8e7e309546fc5fd25db1c0f03487e43a91f909"}
Oct 30 21:30:25.381097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.381035    4810 scope.go:122] "RemoveContainer" containerID="98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e"
Oct 30 21:30:25.381296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.381153    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"
Oct 30 21:30:25.399686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.399662    4810 scope.go:122] "RemoveContainer" containerID="07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73"
Oct 30 21:30:25.410556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.410533    4810 scope.go:122] "RemoveContainer" containerID="98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e"
Oct 30 21:30:25.410894 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:25.410840    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e\": container with ID starting with 98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e not found: ID does not exist" containerID="98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e"
Oct 30 21:30:25.411012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.410909    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e"} err="failed to get container status \"98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e\": rpc error: code = NotFound desc = could not find container \"98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e\": container with ID starting with 98730b2a67cb5af46820312e52bac0967c08fab8e64f97a1405e4207d4cf323e not found: ID does not exist"
Oct 30 21:30:25.411012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.410938    4810 scope.go:122] "RemoveContainer" containerID="07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73"
Oct 30 21:30:25.411220 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:25.411201    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73\": container with ID starting with 07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73 not found: ID does not exist" containerID="07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73"
Oct 30 21:30:25.411261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.411227    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73"} err="failed to get container status \"07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73\": rpc error: code = NotFound desc = could not find container \"07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73\": container with ID starting with 07df7a42911542dc9f8707dde6acfe6879bae5749a545aa0525f2d9c3d300d73 not found: ID does not exist"
Oct 30 21:30:25.451006 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.450981    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d205dba-ff07-4742-8509-44948ba1af95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d205dba-ff07-4742-8509-44948ba1af95-istio-podinfo\") pod \"9d205dba-ff07-4742-8509-44948ba1af95\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") "
Oct 30 21:30:25.451139 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451068    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-credential-socket\") pod \"9d205dba-ff07-4742-8509-44948ba1af95\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") "
Oct 30 21:30:25.451139 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451110    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-envoy\") pod \"9d205dba-ff07-4742-8509-44948ba1af95\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") "
Oct 30 21:30:25.451258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451145    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-kube-api-access-bbcbh\" (UniqueName: \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-kube-api-access-bbcbh\") pod \"9d205dba-ff07-4742-8509-44948ba1af95\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") "
Oct 30 21:30:25.451258 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451250    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-data\") pod \"9d205dba-ff07-4742-8509-44948ba1af95\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") "
Oct 30 21:30:25.451412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451308    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-socket\") pod \"9d205dba-ff07-4742-8509-44948ba1af95\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") "
Oct 30 21:30:25.451412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451376    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-istio-token\" (UniqueName: \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-istio-token\") pod \"9d205dba-ff07-4742-8509-44948ba1af95\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") "
Oct 30 21:30:25.451524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451469    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-certs\") pod \"9d205dba-ff07-4742-8509-44948ba1af95\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") "
Oct 30 21:30:25.451584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451541    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d205dba-ff07-4742-8509-44948ba1af95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d205dba-ff07-4742-8509-44948ba1af95-istiod-ca-cert\") pod \"9d205dba-ff07-4742-8509-44948ba1af95\" (UID: \"9d205dba-ff07-4742-8509-44948ba1af95\") "
Oct 30 21:30:25.451643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451595    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-data" pod "9d205dba-ff07-4742-8509-44948ba1af95" (UID: "9d205dba-ff07-4742-8509-44948ba1af95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:25.451890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451846    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-certs" pod "9d205dba-ff07-4742-8509-44948ba1af95" (UID: "9d205dba-ff07-4742-8509-44948ba1af95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:25.452021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451932    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-socket" pod "9d205dba-ff07-4742-8509-44948ba1af95" (UID: "9d205dba-ff07-4742-8509-44948ba1af95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:25.452021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.451982    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-credential-socket" pod "9d205dba-ff07-4742-8509-44948ba1af95" (UID: "9d205dba-ff07-4742-8509-44948ba1af95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:25.452153 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.452059    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d205dba-ff07-4742-8509-44948ba1af95-istiod-ca-cert" pod "9d205dba-ff07-4742-8509-44948ba1af95" (UID: "9d205dba-ff07-4742-8509-44948ba1af95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:25.452224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.452203    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d205dba-ff07-4742-8509-44948ba1af95-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.452281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.452232    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.452281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.452249    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.452281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.452264    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.452463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.452279    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.453485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.453459    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d205dba-ff07-4742-8509-44948ba1af95-istio-podinfo" pod "9d205dba-ff07-4742-8509-44948ba1af95" (UID: "9d205dba-ff07-4742-8509-44948ba1af95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:30:25.453485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.453472    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-kube-api-access-bbcbh" pod "9d205dba-ff07-4742-8509-44948ba1af95" (UID: "9d205dba-ff07-4742-8509-44948ba1af95"). InnerVolumeSpecName "kube-api-access-bbcbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:25.453621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.453464    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-envoy" pod "9d205dba-ff07-4742-8509-44948ba1af95" (UID: "9d205dba-ff07-4742-8509-44948ba1af95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:25.453678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.453644    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-istio-token" pod "9d205dba-ff07-4742-8509-44948ba1af95" (UID: "9d205dba-ff07-4742-8509-44948ba1af95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:25.553194 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.553165    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d205dba-ff07-4742-8509-44948ba1af95-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.553194 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.553192    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d205dba-ff07-4742-8509-44948ba1af95-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.553399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.553202    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbcbh\" (UniqueName: \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-kube-api-access-bbcbh\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.553399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.553212    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d205dba-ff07-4742-8509-44948ba1af95-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.664119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.664029    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" path="/var/lib/kubelet/pods/112b0e88-a621-4dc9-bf1f-1099de0cfe5d/volumes"
Oct 30 21:30:25.664569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.664548    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612" path="/var/lib/kubelet/pods/1ac65f6a-7beb-4c50-a0ba-546af2072612/volumes"
Oct 30 21:30:25.665036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.665017    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba" path="/var/lib/kubelet/pods/40bb8f64-ee03-4754-89ed-5c69010f4eba/volumes"
Oct 30 21:30:25.665781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.665760    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c97f25-107d-43c7-ad29-52fac553bedc" path="/var/lib/kubelet/pods/64c97f25-107d-43c7-ad29-52fac553bedc/volumes"
Oct 30 21:30:25.666243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.666222    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" path="/var/lib/kubelet/pods/8fd73835-f5e2-47b8-8403-3aa7f03492eb/volumes"
Oct 30 21:30:25.666685 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.666667    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" path="/var/lib/kubelet/pods/cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380/volumes"
Oct 30 21:30:25.702373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.702333    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"]
Oct 30 21:30:25.707439 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:25.707411    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2ac526-openshift-default-845b5867fc-4qxbp"]
Oct 30 21:30:27.664096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:27.664056    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d205dba-ff07-4742-8509-44948ba1af95" path="/var/lib/kubelet/pods/9d205dba-ff07-4742-8509-44948ba1af95/volumes"
Oct 30 21:30:27.772960 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:27.772928    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"]
Oct 30 21:30:27.794877 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:27.788262    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"]
Oct 30 21:30:28.847036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:28.846993    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8" podUID="5b2866f7-5408-4815-b3b3-c898ce73181f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:30.272504 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:30.272469    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"]
Oct 30 21:30:30.292559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:30.292283    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"]
Oct 30 21:30:30.875949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:30.875912    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"]
Oct 30 21:30:31.161267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.161224    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"]
Oct 30 21:30:31.162598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162567    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerName="cleanup"
Oct 30 21:30:31.162598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162594    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162610    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162618    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162635    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162643    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162657    4810 cpu_manager.go:414] "containerMap: removing container" podUID="63a2b983-ff5b-411f-85de-01a6dbae639e" containerName="maas-api"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162664    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a2b983-ff5b-411f-85de-01a6dbae639e" containerName="maas-api"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162674    4810 cpu_manager.go:414] "containerMap: removing container" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162681    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162694    4810 cpu_manager.go:414] "containerMap: removing container" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162701    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162716    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162723    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162735    4810 cpu_manager.go:414] "containerMap: removing container" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162741    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162758    4810 cpu_manager.go:414] "containerMap: removing container" podUID="052961e9-c81b-4273-94b9-4704b21dba4f" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162765    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052961e9-c81b-4273-94b9-4704b21dba4f" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162774    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162782    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162792    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerName="cleanup"
Oct 30 21:30:31.162791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162799    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162815    4810 cpu_manager.go:414] "containerMap: removing container" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162822    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162833    4810 cpu_manager.go:414] "containerMap: removing container" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162841    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162850    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162904    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162917    4810 cpu_manager.go:414] "containerMap: removing container" podUID="daad11ef-c385-4cf8-975b-e63912445add" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162924    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daad11ef-c385-4cf8-975b-e63912445add" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162952    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162959    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162974    4810 cpu_manager.go:414] "containerMap: removing container" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162980    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162989    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.162997    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163007    4810 cpu_manager.go:414] "containerMap: removing container" podUID="aca678bf-105c-472c-90e8-3226bdde7e25" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163014    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca678bf-105c-472c-90e8-3226bdde7e25" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163072    4810 cpu_manager.go:414] "containerMap: removing container" podUID="35a5aa95-faba-4af6-a80e-137745490e73" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163081    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a5aa95-faba-4af6-a80e-137745490e73" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163092    4810 cpu_manager.go:414] "containerMap: removing container" podUID="daad11ef-c385-4cf8-975b-e63912445add" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163100    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daad11ef-c385-4cf8-975b-e63912445add" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163110    4810 cpu_manager.go:414] "containerMap: removing container" podUID="aca678bf-105c-472c-90e8-3226bdde7e25" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163117    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca678bf-105c-472c-90e8-3226bdde7e25" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163131    4810 cpu_manager.go:414] "containerMap: removing container" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163138    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163147    4810 cpu_manager.go:414] "containerMap: removing container" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163153    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163167    4810 cpu_manager.go:414] "containerMap: removing container" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163174    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163183    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163191    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163206    4810 cpu_manager.go:414] "containerMap: removing container" podUID="64c97f25-107d-43c7-ad29-52fac553bedc" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163213    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c97f25-107d-43c7-ad29-52fac553bedc" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163235    4810 cpu_manager.go:414] "containerMap: removing container" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163243    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163254    4810 cpu_manager.go:414] "containerMap: removing container" podUID="b92d5aab-dbcf-47e0-a2c9-111af9c207c6" containerName="payload-pre-processing"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163262    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92d5aab-dbcf-47e0-a2c9-111af9c207c6" containerName="payload-pre-processing"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163275    4810 cpu_manager.go:414] "containerMap: removing container" podUID="64c97f25-107d-43c7-ad29-52fac553bedc" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163281    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c97f25-107d-43c7-ad29-52fac553bedc" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163310    4810 cpu_manager.go:414] "containerMap: removing container" podUID="33b2eef0-0705-463e-9c89-96f86d0c1576" containerName="payload-processing"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163318    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b2eef0-0705-463e-9c89-96f86d0c1576" containerName="payload-processing"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163364    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163372    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163388    4810 cpu_manager.go:414] "containerMap: removing container" podUID="052961e9-c81b-4273-94b9-4704b21dba4f" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163395    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052961e9-c81b-4273-94b9-4704b21dba4f" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163423    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163430    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163445    4810 cpu_manager.go:414] "containerMap: removing container" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163451    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163461    4810 cpu_manager.go:414] "containerMap: removing container" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163467    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163497    4810 cpu_manager.go:414] "containerMap: removing container" podUID="6b22d659-57de-4142-8da1-8b0705a58fc0" containerName="payload-processing"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163504    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b22d659-57de-4142-8da1-8b0705a58fc0" containerName="payload-processing"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163528    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163535    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163545    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163552    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163563    4810 cpu_manager.go:414] "containerMap: removing container" podUID="35a5aa95-faba-4af6-a80e-137745490e73" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163571    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a5aa95-faba-4af6-a80e-137745490e73" containerName="cleanup"
Oct 30 21:30:31.163849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.163757    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164749    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c97f25-107d-43c7-ad29-52fac553bedc" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164770    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164781    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164791    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164809    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b22d659-57de-4142-8da1-8b0705a58fc0" containerName="payload-processing"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164820    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="daad11ef-c385-4cf8-975b-e63912445add" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164831    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164839    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d205dba-ff07-4742-8509-44948ba1af95" containerName="istio-proxy"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164852    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a5aa95-faba-4af6-a80e-137745490e73" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164876    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164883    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164893    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164901    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a2b983-ff5b-411f-85de-01a6dbae639e" containerName="maas-api"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164909    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerName="cleanup"
Oct 30 21:30:31.168949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.164997    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174390    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174429    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174441    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="052961e9-c81b-4273-94b9-4704b21dba4f" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174450    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174460    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174472    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174482    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174499    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="daad11ef-c385-4cf8-975b-e63912445add" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174512    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174522    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174529    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca678bf-105c-472c-90e8-3226bdde7e25" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174539    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="daad11ef-c385-4cf8-975b-e63912445add" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174547    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca678bf-105c-472c-90e8-3226bdde7e25" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174557    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf881a3-86a2-494d-b8e5-3a789801bd41" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174566    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="052961e9-c81b-4273-94b9-4704b21dba4f" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174574    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174590    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174598    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174607    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c97f25-107d-43c7-ad29-52fac553bedc" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174616    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e54b96-7b3a-421c-9ca0-312047e92c64" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174626    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92d5aab-dbcf-47e0-a2c9-111af9c207c6" containerName="payload-pre-processing"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174635    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c97f25-107d-43c7-ad29-52fac553bedc" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174649    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b2eef0-0705-463e-9c89-96f86d0c1576" containerName="payload-processing"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174662    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174670    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a5aa95-faba-4af6-a80e-137745490e73" containerName="cleanup"
Oct 30 21:30:31.177091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.174678    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="052961e9-c81b-4273-94b9-4704b21dba4f" containerName="cleanup"
Oct 30 21:30:31.204200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.199224    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"]
Oct 30 21:30:31.204200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.199344    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.207708 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.206666    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:30:31.207708 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.207124    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:30:31.262060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.262033    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68r9r\" (UniqueName: \"kubernetes.io/projected/575e3275-23c3-4643-8b30-e5922326f280-kube-api-access-68r9r\") pod \"maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.262206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.262074    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575e3275-23c3-4643-8b30-e5922326f280-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.262206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.262180    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575e3275-23c3-4643-8b30-e5922326f280-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.264930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.264907    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:30:31.363441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.363343    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b2866f7-5408-4815-b3b3-c898ce73181f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2866f7-5408-4815-b3b3-c898ce73181f-plugins-config-volume\") pod \"5b2866f7-5408-4815-b3b3-c898ce73181f\" (UID: \"5b2866f7-5408-4815-b3b3-c898ce73181f\") "
Oct 30 21:30:31.363970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.363634    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b2866f7-5408-4815-b3b3-c898ce73181f-kube-api-access-v5wh2\" (UniqueName: \"kubernetes.io/projected/5b2866f7-5408-4815-b3b3-c898ce73181f-kube-api-access-v5wh2\") pod \"5b2866f7-5408-4815-b3b3-c898ce73181f\" (UID: \"5b2866f7-5408-4815-b3b3-c898ce73181f\") "
Oct 30 21:30:31.363970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.363800    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b2866f7-5408-4815-b3b3-c898ce73181f-plugins-config-volume" pod "5b2866f7-5408-4815-b3b3-c898ce73181f" (UID: "5b2866f7-5408-4815-b3b3-c898ce73181f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:31.364071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.364014    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68r9r\" (UniqueName: \"kubernetes.io/projected/575e3275-23c3-4643-8b30-e5922326f280-kube-api-access-68r9r\") pod \"maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.364134 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.364063    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575e3275-23c3-4643-8b30-e5922326f280-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.364213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.364196    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575e3275-23c3-4643-8b30-e5922326f280-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.364389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.364372    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2866f7-5408-4815-b3b3-c898ce73181f-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:31.365156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.365022    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575e3275-23c3-4643-8b30-e5922326f280-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.368289 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.368259    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2866f7-5408-4815-b3b3-c898ce73181f-kube-api-access-v5wh2" pod "5b2866f7-5408-4815-b3b3-c898ce73181f" (UID: "5b2866f7-5408-4815-b3b3-c898ce73181f"). InnerVolumeSpecName "kube-api-access-v5wh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:31.370246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.370221    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575e3275-23c3-4643-8b30-e5922326f280-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.373390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.373364    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68r9r\" (UniqueName: \"kubernetes.io/projected/575e3275-23c3-4643-8b30-e5922326f280-kube-api-access-68r9r\") pod \"maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.469999 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.468707    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5wh2\" (UniqueName: \"kubernetes.io/projected/5b2866f7-5408-4815-b3b3-c898ce73181f-kube-api-access-v5wh2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:31.505909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.503797    4810 generic.go:356] "Generic (PLEG): container finished" podID="5b2866f7-5408-4815-b3b3-c898ce73181f" containerID="b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4" exitCode=0
Oct 30 21:30:31.505909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.504166    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8" event={"ID":"5b2866f7-5408-4815-b3b3-c898ce73181f","Type":"ContainerDied","Data":"b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4"}
Oct 30 21:30:31.505909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.504221    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8" event={"ID":"5b2866f7-5408-4815-b3b3-c898ce73181f","Type":"ContainerDied","Data":"6771d717ab17d272daff4413344a316f01237475857a54d972ad5e076622a90b"}
Oct 30 21:30:31.505909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.504252    4810 scope.go:122] "RemoveContainer" containerID="b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4"
Oct 30 21:30:31.505909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.504417    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"
Oct 30 21:30:31.532127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.532052    4810 scope.go:122] "RemoveContainer" containerID="b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4"
Oct 30 21:30:31.532632 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:31.532574    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4\": container with ID starting with b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4 not found: ID does not exist" containerID="b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4"
Oct 30 21:30:31.532714 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.532643    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4"} err="failed to get container status \"b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4\": rpc error: code = NotFound desc = could not find container \"b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4\": container with ID starting with b9eb9d7f3d8e2ab7bc8aaf9bde93f2daa4862dca13b9fdfcedb9a0a4223210f4 not found: ID does not exist"
Oct 30 21:30:31.551525 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.551496    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"]
Oct 30 21:30:31.555533 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.555510    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2ac526-8659cfcc6c-mpnc8"]
Oct 30 21:30:31.561405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.561386    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:31.666318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.666289    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b2866f7-5408-4815-b3b3-c898ce73181f" path="/var/lib/kubelet/pods/5b2866f7-5408-4815-b3b3-c898ce73181f/volumes"
Oct 30 21:30:31.853329 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:31.853299    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575e3275_23c3_4643_8b30_e5922326f280.slice/crio-db01a3e1433a954cf434699b33b4f1db7bc7204704dbb29c451559f57aef25a5 WatchSource:0}: Error finding container db01a3e1433a954cf434699b33b4f1db7bc7204704dbb29c451559f57aef25a5: Status 404 returned error can't find the container with id db01a3e1433a954cf434699b33b4f1db7bc7204704dbb29c451559f57aef25a5
Oct 30 21:30:31.853516 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:31.853495    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"]
Oct 30 21:30:32.516749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:32.516651    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn" event={"ID":"575e3275-23c3-4643-8b30-e5922326f280","Type":"ContainerStarted","Data":"1b46aa92d22767a0feb317e48d920031623388bdb3b46649c3080a8bc44e02f3"}
Oct 30 21:30:32.516749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:32.516722    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn" event={"ID":"575e3275-23c3-4643-8b30-e5922326f280","Type":"ContainerStarted","Data":"db01a3e1433a954cf434699b33b4f1db7bc7204704dbb29c451559f57aef25a5"}
Oct 30 21:30:32.538208 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:32.538154    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn" podStartSLOduration=1.538141171 podStartE2EDuration="1.538141171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:32.535901447 +0000 UTC m=+3199.027717995" watchObservedRunningTime="2026-10-30 21:30:32.538141171 +0000 UTC m=+3199.029957714"
Oct 30 21:30:32.773490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:32.773448    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv" podUID="96d8eef0-e5c5-4f87-9502-973ef001a54e" containerName="payload-processing" containerID="cri-o://6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323" gracePeriod=25
Oct 30 21:30:32.790946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:32.790915    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d" podUID="3a804f13-7cd2-4949-bd3d-ecf594465e0b" containerName="payload-pre-processing" containerID="cri-o://f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934" gracePeriod=25
Oct 30 21:30:33.073758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.073732    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:30:33.200703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.200666    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d8eef0-e5c5-4f87-9502-973ef001a54e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d8eef0-e5c5-4f87-9502-973ef001a54e-plugins-config-volume\") pod \"96d8eef0-e5c5-4f87-9502-973ef001a54e\" (UID: \"96d8eef0-e5c5-4f87-9502-973ef001a54e\") "
Oct 30 21:30:33.200917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.200847    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d8eef0-e5c5-4f87-9502-973ef001a54e-kube-api-access-nlzn2\" (UniqueName: \"kubernetes.io/projected/96d8eef0-e5c5-4f87-9502-973ef001a54e-kube-api-access-nlzn2\") pod \"96d8eef0-e5c5-4f87-9502-973ef001a54e\" (UID: \"96d8eef0-e5c5-4f87-9502-973ef001a54e\") "
Oct 30 21:30:33.201147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.201118    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d8eef0-e5c5-4f87-9502-973ef001a54e-plugins-config-volume" pod "96d8eef0-e5c5-4f87-9502-973ef001a54e" (UID: "96d8eef0-e5c5-4f87-9502-973ef001a54e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:33.201678 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.201653    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d8eef0-e5c5-4f87-9502-973ef001a54e-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:33.203354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.203321    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d8eef0-e5c5-4f87-9502-973ef001a54e-kube-api-access-nlzn2" pod "96d8eef0-e5c5-4f87-9502-973ef001a54e" (UID: "96d8eef0-e5c5-4f87-9502-973ef001a54e"). InnerVolumeSpecName "kube-api-access-nlzn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:33.303413 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.303342    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzn2\" (UniqueName: \"kubernetes.io/projected/96d8eef0-e5c5-4f87-9502-973ef001a54e-kube-api-access-nlzn2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:33.528188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.528154    4810 generic.go:356] "Generic (PLEG): container finished" podID="575e3275-23c3-4643-8b30-e5922326f280" containerID="1b46aa92d22767a0feb317e48d920031623388bdb3b46649c3080a8bc44e02f3" exitCode=0
Oct 30 21:30:33.528671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.528250    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn" event={"ID":"575e3275-23c3-4643-8b30-e5922326f280","Type":"ContainerDied","Data":"1b46aa92d22767a0feb317e48d920031623388bdb3b46649c3080a8bc44e02f3"}
Oct 30 21:30:33.529651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.529632    4810 generic.go:356] "Generic (PLEG): container finished" podID="96d8eef0-e5c5-4f87-9502-973ef001a54e" containerID="6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323" exitCode=0
Oct 30 21:30:33.529719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.529673    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"
Oct 30 21:30:33.529763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.529713    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv" event={"ID":"96d8eef0-e5c5-4f87-9502-973ef001a54e","Type":"ContainerDied","Data":"6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323"}
Oct 30 21:30:33.529763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.529756    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv" event={"ID":"96d8eef0-e5c5-4f87-9502-973ef001a54e","Type":"ContainerDied","Data":"acdbdf5579639d12322895fda1ad2498a1785cca140edeb001795f67847d5e73"}
Oct 30 21:30:33.529836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.529778    4810 scope.go:122] "RemoveContainer" containerID="6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323"
Oct 30 21:30:33.541666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.541652    4810 scope.go:122] "RemoveContainer" containerID="6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323"
Oct 30 21:30:33.541927 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:33.541911    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323\": container with ID starting with 6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323 not found: ID does not exist" containerID="6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323"
Oct 30 21:30:33.541995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.541935    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323"} err="failed to get container status \"6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323\": rpc error: code = NotFound desc = could not find container \"6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323\": container with ID starting with 6f22b2494ac3d0c899ff8b437b58931a80def0e861f3c0cd3cc62af60a67e323 not found: ID does not exist"
Oct 30 21:30:33.560179 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.560120    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"]
Oct 30 21:30:33.566357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.566329    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b78e4-7fdbd787bf-jmljv"]
Oct 30 21:30:33.666502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:33.666463    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d8eef0-e5c5-4f87-9502-973ef001a54e" path="/var/lib/kubelet/pods/96d8eef0-e5c5-4f87-9502-973ef001a54e/volumes"
Oct 30 21:30:34.754901 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.754831    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6" podUID="d48ab868-05de-46c7-89ba-14468f8d778b" containerName="main" containerID="cri-o://0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3" gracePeriod=45
Oct 30 21:30:34.767718 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.767696    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:34.837922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.837802    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575e3275-23c3-4643-8b30-e5922326f280-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575e3275-23c3-4643-8b30-e5922326f280-maas-api-openshift-ca\") pod \"575e3275-23c3-4643-8b30-e5922326f280\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") "
Oct 30 21:30:34.838111 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.837989    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575e3275-23c3-4643-8b30-e5922326f280-kube-api-access-68r9r\" (UniqueName: \"kubernetes.io/projected/575e3275-23c3-4643-8b30-e5922326f280-kube-api-access-68r9r\") pod \"575e3275-23c3-4643-8b30-e5922326f280\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") "
Oct 30 21:30:34.838111 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.838056    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/575e3275-23c3-4643-8b30-e5922326f280-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575e3275-23c3-4643-8b30-e5922326f280-maas-api-xks-ca\") pod \"575e3275-23c3-4643-8b30-e5922326f280\" (UID: \"575e3275-23c3-4643-8b30-e5922326f280\") "
Oct 30 21:30:34.838279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.838246    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575e3275-23c3-4643-8b30-e5922326f280-maas-api-openshift-ca" pod "575e3275-23c3-4643-8b30-e5922326f280" (UID: "575e3275-23c3-4643-8b30-e5922326f280"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:34.838919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.838891    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/575e3275-23c3-4643-8b30-e5922326f280-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:34.840381 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.840352    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575e3275-23c3-4643-8b30-e5922326f280-kube-api-access-68r9r" pod "575e3275-23c3-4643-8b30-e5922326f280" (UID: "575e3275-23c3-4643-8b30-e5922326f280"). InnerVolumeSpecName "kube-api-access-68r9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:34.840655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.840633    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575e3275-23c3-4643-8b30-e5922326f280-maas-api-xks-ca" pod "575e3275-23c3-4643-8b30-e5922326f280" (UID: "575e3275-23c3-4643-8b30-e5922326f280"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:34.939957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.939924    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68r9r\" (UniqueName: \"kubernetes.io/projected/575e3275-23c3-4643-8b30-e5922326f280-kube-api-access-68r9r\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:34.939957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:34.939959    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/575e3275-23c3-4643-8b30-e5922326f280-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:35.020553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.020520    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:30:35.143089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.143056    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-dshm\") pod \"d48ab868-05de-46c7-89ba-14468f8d778b\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") "
Oct 30 21:30:35.143294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.143178    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48ab868-05de-46c7-89ba-14468f8d778b-kube-api-access-8npfr\" (UniqueName: \"kubernetes.io/projected/d48ab868-05de-46c7-89ba-14468f8d778b-kube-api-access-8npfr\") pod \"d48ab868-05de-46c7-89ba-14468f8d778b\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") "
Oct 30 21:30:35.143294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.143244    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48ab868-05de-46c7-89ba-14468f8d778b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d48ab868-05de-46c7-89ba-14468f8d778b-tls-certs\") pod \"d48ab868-05de-46c7-89ba-14468f8d778b\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") "
Oct 30 21:30:35.143294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.143282    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-model-cache\") pod \"d48ab868-05de-46c7-89ba-14468f8d778b\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") "
Oct 30 21:30:35.143484 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.143317    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-tmp-dir\") pod \"d48ab868-05de-46c7-89ba-14468f8d778b\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") "
Oct 30 21:30:35.143484 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.143348    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-home\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-home\") pod \"d48ab868-05de-46c7-89ba-14468f8d778b\" (UID: \"d48ab868-05de-46c7-89ba-14468f8d778b\") "
Oct 30 21:30:35.143601 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.143538    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-model-cache" pod "d48ab868-05de-46c7-89ba-14468f8d778b" (UID: "d48ab868-05de-46c7-89ba-14468f8d778b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:35.143657 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.143617    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-tmp-dir" pod "d48ab868-05de-46c7-89ba-14468f8d778b" (UID: "d48ab868-05de-46c7-89ba-14468f8d778b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:35.143729 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.143706    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-home" pod "d48ab868-05de-46c7-89ba-14468f8d778b" (UID: "d48ab868-05de-46c7-89ba-14468f8d778b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:35.144029 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.144006    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:35.144135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.144035    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:35.144135 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.144051    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:35.145599 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.145575    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-dshm" pod "d48ab868-05de-46c7-89ba-14468f8d778b" (UID: "d48ab868-05de-46c7-89ba-14468f8d778b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:35.145599 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.145585    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48ab868-05de-46c7-89ba-14468f8d778b-tls-certs" pod "d48ab868-05de-46c7-89ba-14468f8d778b" (UID: "d48ab868-05de-46c7-89ba-14468f8d778b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:35.145719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.145640    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48ab868-05de-46c7-89ba-14468f8d778b-kube-api-access-8npfr" pod "d48ab868-05de-46c7-89ba-14468f8d778b" (UID: "d48ab868-05de-46c7-89ba-14468f8d778b"). InnerVolumeSpecName "kube-api-access-8npfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:35.245284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.245249    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48ab868-05de-46c7-89ba-14468f8d778b-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:35.245284 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.245281    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8npfr\" (UniqueName: \"kubernetes.io/projected/d48ab868-05de-46c7-89ba-14468f8d778b-kube-api-access-8npfr\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:35.245502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.245295    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48ab868-05de-46c7-89ba-14468f8d778b-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:35.273133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.273052    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs" podUID="275a0d03-1705-464c-aff5-4b5b8fb668fe" containerName="payload-processing" containerID="cri-o://850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953" gracePeriod=25
Oct 30 21:30:35.293276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.293234    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz" podUID="029c21db-1fbc-4a68-a513-1fc9da53d370" containerName="payload-pre-processing" containerID="cri-o://4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783" gracePeriod=25
Oct 30 21:30:35.382670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.382633    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:30:35.553129 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.553046    4810 generic.go:356] "Generic (PLEG): container finished" podID="d48ab868-05de-46c7-89ba-14468f8d778b" containerID="0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3" exitCode=0
Oct 30 21:30:35.553129 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.553095    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"
Oct 30 21:30:35.553338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.553139    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6" event={"ID":"d48ab868-05de-46c7-89ba-14468f8d778b","Type":"ContainerDied","Data":"0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3"}
Oct 30 21:30:35.553338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.553191    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6" event={"ID":"d48ab868-05de-46c7-89ba-14468f8d778b","Type":"ContainerDied","Data":"1f1bbe2690423c459556dfa27db7fe7454a8b5bd9ea9b8ac39766bd16108bab7"}
Oct 30 21:30:35.553338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.553216    4810 scope.go:122] "RemoveContainer" containerID="0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3"
Oct 30 21:30:35.555173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.555043    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn" event={"ID":"575e3275-23c3-4643-8b30-e5922326f280","Type":"ContainerDied","Data":"db01a3e1433a954cf434699b33b4f1db7bc7204704dbb29c451559f57aef25a5"}
Oct 30 21:30:35.555173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.555080    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"
Oct 30 21:30:35.555173 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.555103    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db01a3e1433a954cf434699b33b4f1db7bc7204704dbb29c451559f57aef25a5"
Oct 30 21:30:35.565839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.565779    4810 scope.go:122] "RemoveContainer" containerID="0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3"
Oct 30 21:30:35.566180 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:35.566152    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3\": container with ID starting with 0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3 not found: ID does not exist" containerID="0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3"
Oct 30 21:30:35.566280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.566191    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3"} err="failed to get container status \"0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3\": rpc error: code = NotFound desc = could not find container \"0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3\": container with ID starting with 0fbb20457c9fca8b3c5969caed53cc887c25789b6d469874a314c0f497eb0ca3 not found: ID does not exist"
Oct 30 21:30:35.583640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.583614    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"]
Oct 30 21:30:35.591247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.591222    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ddcf93/test-model-ddcf93-kserve-b98c95d45-n4ng6"]
Oct 30 21:30:35.672780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:35.672739    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48ab868-05de-46c7-89ba-14468f8d778b" path="/var/lib/kubelet/pods/d48ab868-05de-46c7-89ba-14468f8d778b/volumes"
Oct 30 21:30:39.765807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:39.765772    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d" podUID="3a804f13-7cd2-4949-bd3d-ecf594465e0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:41.029050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:41.029010    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"]
Oct 30 21:30:41.032826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:41.032798    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5ff4a45-qt5rp"]
Oct 30 21:30:41.255828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:41.255789    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"]
Oct 30 21:30:41.274884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:41.274706    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"]
Oct 30 21:30:41.666189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:41.666148    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82da14a4-2ad0-4915-a4ac-4e46a32d5586" path="/var/lib/kubelet/pods/82da14a4-2ad0-4915-a4ac-4e46a32d5586/volumes"
Oct 30 21:30:43.172002 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:43.171954    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs" podUID="275a0d03-1705-464c-aff5-4b5b8fb668fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:43.172501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:43.172131    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz" podUID="029c21db-1fbc-4a68-a513-1fc9da53d370" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:43.337711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:43.337675    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"]
Oct 30 21:30:43.337951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:43.337921    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q" podUID="8445b211-0a2f-484d-bfe8-caf789fd035b" containerName="maas-api" containerID="cri-o://a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741" gracePeriod=30
Oct 30 21:30:43.727456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:43.727327    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"]
Oct 30 21:30:43.731640 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:43.731610    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddcf93-hq4kn"]
Oct 30 21:30:43.910630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:43.910594    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:30:44.001717 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.001682    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-maas-api-tls\") pod \"8445b211-0a2f-484d-bfe8-caf789fd035b\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") "
Oct 30 21:30:44.001914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.001737    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8445b211-0a2f-484d-bfe8-caf789fd035b-kube-api-access-hdvl2\" (UniqueName: \"kubernetes.io/projected/8445b211-0a2f-484d-bfe8-caf789fd035b-kube-api-access-hdvl2\") pod \"8445b211-0a2f-484d-bfe8-caf789fd035b\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") "
Oct 30 21:30:44.001914 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.001896    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8445b211-0a2f-484d-bfe8-caf789fd035b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8445b211-0a2f-484d-bfe8-caf789fd035b-openshift-service-ca\") pod \"8445b211-0a2f-484d-bfe8-caf789fd035b\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") "
Oct 30 21:30:44.002062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.001947    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-metrics-certs\") pod \"8445b211-0a2f-484d-bfe8-caf789fd035b\" (UID: \"8445b211-0a2f-484d-bfe8-caf789fd035b\") "
Oct 30 21:30:44.002320 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.002291    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8445b211-0a2f-484d-bfe8-caf789fd035b-openshift-service-ca" pod "8445b211-0a2f-484d-bfe8-caf789fd035b" (UID: "8445b211-0a2f-484d-bfe8-caf789fd035b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:44.002539 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.002517    4810 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8445b211-0a2f-484d-bfe8-caf789fd035b-openshift-service-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:44.004449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.004422    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-maas-api-tls" pod "8445b211-0a2f-484d-bfe8-caf789fd035b" (UID: "8445b211-0a2f-484d-bfe8-caf789fd035b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:44.004552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.004528    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8445b211-0a2f-484d-bfe8-caf789fd035b-kube-api-access-hdvl2" pod "8445b211-0a2f-484d-bfe8-caf789fd035b" (UID: "8445b211-0a2f-484d-bfe8-caf789fd035b"). InnerVolumeSpecName "kube-api-access-hdvl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:44.004605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.004578    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-metrics-certs" pod "8445b211-0a2f-484d-bfe8-caf789fd035b" (UID: "8445b211-0a2f-484d-bfe8-caf789fd035b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:44.104314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.104272    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-maas-api-tls\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:44.104314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.104312    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdvl2\" (UniqueName: \"kubernetes.io/projected/8445b211-0a2f-484d-bfe8-caf789fd035b-kube-api-access-hdvl2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:44.104314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.104327    4810 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8445b211-0a2f-484d-bfe8-caf789fd035b-metrics-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:44.678486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.678449    4810 generic.go:356] "Generic (PLEG): container finished" podID="8445b211-0a2f-484d-bfe8-caf789fd035b" containerID="a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741" exitCode=0
Oct 30 21:30:44.679151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.678501    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"
Oct 30 21:30:44.679151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.678537    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q" event={"ID":"8445b211-0a2f-484d-bfe8-caf789fd035b","Type":"ContainerDied","Data":"a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741"}
Oct 30 21:30:44.679151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.678607    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q" event={"ID":"8445b211-0a2f-484d-bfe8-caf789fd035b","Type":"ContainerDied","Data":"25f089a53cf9b13e662f8accc7529c547ffb9f7df45144758546118376debd02"}
Oct 30 21:30:44.679151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.678635    4810 scope.go:122] "RemoveContainer" containerID="a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741"
Oct 30 21:30:44.693784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.693762    4810 scope.go:122] "RemoveContainer" containerID="a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741"
Oct 30 21:30:44.694142 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:44.694114    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741\": container with ID starting with a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741 not found: ID does not exist" containerID="a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741"
Oct 30 21:30:44.694275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.694147    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741"} err="failed to get container status \"a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741\": rpc error: code = NotFound desc = could not find container \"a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741\": container with ID starting with a4820c84d9512e1ab72a036ae7dd8a89e5bcebf62abaa186c80f651643a45741 not found: ID does not exist"
Oct 30 21:30:44.706784 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.706757    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"]
Oct 30 21:30:44.718016 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:44.717992    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddcf93-9ff7f5b4c-b2g2q"]
Oct 30 21:30:45.663613 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:45.663584    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575e3275-23c3-4643-8b30-e5922326f280" path="/var/lib/kubelet/pods/575e3275-23c3-4643-8b30-e5922326f280/volumes"
Oct 30 21:30:45.664047 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:45.664033    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8445b211-0a2f-484d-bfe8-caf789fd035b" path="/var/lib/kubelet/pods/8445b211-0a2f-484d-bfe8-caf789fd035b/volumes"
Oct 30 21:30:46.065808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.065701    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt" podUID="a7e8d92d-252b-4abb-ab86-3efc8c0759a1" containerName="main" containerID="cri-o://93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c" gracePeriod=45
Oct 30 21:30:46.257140 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.257098    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24" podUID="cb523c54-0274-45d9-960d-b395e30205f1" containerName="payload-processing" containerID="cri-o://3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a" gracePeriod=25
Oct 30 21:30:46.279808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.276258    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk" podUID="95743c31-995c-4ff4-b66c-174478a72987" containerName="payload-pre-processing" containerID="cri-o://40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f" gracePeriod=25
Oct 30 21:30:46.355049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.355024    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:30:46.435429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.435396    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tmp-dir\") pod \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") "
Oct 30 21:30:46.435615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.435519    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-home\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-home\") pod \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") "
Oct 30 21:30:46.435615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.435543    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-model-cache\") pod \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") "
Oct 30 21:30:46.435615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.435605    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-dshm\") pod \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") "
Oct 30 21:30:46.435818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.435666    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tls-certs\") pod \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") "
Oct 30 21:30:46.435818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.435756    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-home" pod "a7e8d92d-252b-4abb-ab86-3efc8c0759a1" (UID: "a7e8d92d-252b-4abb-ab86-3efc8c0759a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:46.435818 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.435797    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-kube-api-access-zzt6p\" (UniqueName: \"kubernetes.io/projected/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-kube-api-access-zzt6p\") pod \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\" (UID: \"a7e8d92d-252b-4abb-ab86-3efc8c0759a1\") "
Oct 30 21:30:46.436042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.435871    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-model-cache" pod "a7e8d92d-252b-4abb-ab86-3efc8c0759a1" (UID: "a7e8d92d-252b-4abb-ab86-3efc8c0759a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:46.436131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.436095    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tmp-dir" pod "a7e8d92d-252b-4abb-ab86-3efc8c0759a1" (UID: "a7e8d92d-252b-4abb-ab86-3efc8c0759a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:46.436425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.436377    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.436425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.436397    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.436560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.436437    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.438195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.438166    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-kube-api-access-zzt6p" pod "a7e8d92d-252b-4abb-ab86-3efc8c0759a1" (UID: "a7e8d92d-252b-4abb-ab86-3efc8c0759a1"). InnerVolumeSpecName "kube-api-access-zzt6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:46.438368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.438345    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-dshm" pod "a7e8d92d-252b-4abb-ab86-3efc8c0759a1" (UID: "a7e8d92d-252b-4abb-ab86-3efc8c0759a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:46.438421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.438351    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tls-certs" pod "a7e8d92d-252b-4abb-ab86-3efc8c0759a1" (UID: "a7e8d92d-252b-4abb-ab86-3efc8c0759a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:46.537651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.537622    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzt6p\" (UniqueName: \"kubernetes.io/projected/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-kube-api-access-zzt6p\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.537651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.537650    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.537832 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.537661    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e8d92d-252b-4abb-ab86-3efc8c0759a1-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.698118 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.698086    4810 generic.go:356] "Generic (PLEG): container finished" podID="a7e8d92d-252b-4abb-ab86-3efc8c0759a1" containerID="93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c" exitCode=0
Oct 30 21:30:46.698328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.698148    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"
Oct 30 21:30:46.698328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.698170    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt" event={"ID":"a7e8d92d-252b-4abb-ab86-3efc8c0759a1","Type":"ContainerDied","Data":"93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c"}
Oct 30 21:30:46.698328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.698214    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt" event={"ID":"a7e8d92d-252b-4abb-ab86-3efc8c0759a1","Type":"ContainerDied","Data":"6e4b848231d8f6e61b30a565b966b1fd36deb5fa8e97e15a4721b649107d887b"}
Oct 30 21:30:46.698328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.698243    4810 scope.go:122] "RemoveContainer" containerID="93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c"
Oct 30 21:30:46.714634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.714617    4810 scope.go:122] "RemoveContainer" containerID="93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c"
Oct 30 21:30:46.714929 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:46.714908    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c\": container with ID starting with 93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c not found: ID does not exist" containerID="93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c"
Oct 30 21:30:46.715022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.714937    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c"} err="failed to get container status \"93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c\": rpc error: code = NotFound desc = could not find container \"93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c\": container with ID starting with 93184b8697956c875d580fb6fd9a64404739541b0cbad97056aa680b5272ac3c not found: ID does not exist"
Oct 30 21:30:46.725410 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.725385    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"]
Oct 30 21:30:46.731738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:46.731714    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5506f4/test-model-5506f4-kserve-658584bdc9-vwqxt"]
Oct 30 21:30:47.664393 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.664361    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e8d92d-252b-4abb-ab86-3efc8c0759a1" path="/var/lib/kubelet/pods/a7e8d92d-252b-4abb-ab86-3efc8c0759a1/volumes"
Oct 30 21:30:47.952257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.952180    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"]
Oct 30 21:30:47.952928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.952911    4810 cpu_manager.go:414] "containerMap: removing container" podUID="64c97f25-107d-43c7-ad29-52fac553bedc" containerName="cleanup"
Oct 30 21:30:47.952928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.952928    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c97f25-107d-43c7-ad29-52fac553bedc" containerName="cleanup"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.952950    4810 cpu_manager.go:414] "containerMap: removing container" podUID="052961e9-c81b-4273-94b9-4704b21dba4f" containerName="cleanup"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.952955    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052961e9-c81b-4273-94b9-4704b21dba4f" containerName="cleanup"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.952967    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8445b211-0a2f-484d-bfe8-caf789fd035b" containerName="maas-api"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.952973    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8445b211-0a2f-484d-bfe8-caf789fd035b" containerName="maas-api"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.952982    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d48ab868-05de-46c7-89ba-14468f8d778b" containerName="main"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.952987    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48ab868-05de-46c7-89ba-14468f8d778b" containerName="main"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.952998    4810 cpu_manager.go:414] "containerMap: removing container" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerName="cleanup"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953002    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerName="cleanup"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953019    4810 cpu_manager.go:414] "containerMap: removing container" podUID="575e3275-23c3-4643-8b30-e5922326f280" containerName="revoke-keys"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953024    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575e3275-23c3-4643-8b30-e5922326f280" containerName="revoke-keys"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953032    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a7e8d92d-252b-4abb-ab86-3efc8c0759a1" containerName="main"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953036    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e8d92d-252b-4abb-ab86-3efc8c0759a1" containerName="main"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953044    4810 cpu_manager.go:414] "containerMap: removing container" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerName="cleanup"
Oct 30 21:30:47.953046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953049    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953062    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953066    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953075    4810 cpu_manager.go:414] "containerMap: removing container" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953079    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112b0e88-a621-4dc9-bf1f-1099de0cfe5d" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953084    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953088    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953098    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5b2866f7-5408-4815-b3b3-c898ce73181f" containerName="payload-pre-processing"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953102    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b2866f7-5408-4815-b3b3-c898ce73181f" containerName="payload-pre-processing"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953109    4810 cpu_manager.go:414] "containerMap: removing container" podUID="96d8eef0-e5c5-4f87-9502-973ef001a54e" containerName="payload-processing"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953113    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d8eef0-e5c5-4f87-9502-973ef001a54e" containerName="payload-processing"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953122    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953126    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953133    4810 cpu_manager.go:414] "containerMap: removing container" podUID="35a5aa95-faba-4af6-a80e-137745490e73" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953137    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a5aa95-faba-4af6-a80e-137745490e73" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953145    4810 cpu_manager.go:414] "containerMap: removing container" podUID="daad11ef-c385-4cf8-975b-e63912445add" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953151    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daad11ef-c385-4cf8-975b-e63912445add" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953156    4810 cpu_manager.go:414] "containerMap: removing container" podUID="aca678bf-105c-472c-90e8-3226bdde7e25" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953160    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca678bf-105c-472c-90e8-3226bdde7e25" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953257    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="575e3275-23c3-4643-8b30-e5922326f280" containerName="revoke-keys"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953265    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e8d92d-252b-4abb-ab86-3efc8c0759a1" containerName="main"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953271    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a4e9f94-c578-43a7-9f01-937d5cc8c679" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953280    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb66d6d2-a9aa-4d56-afc0-c6ecd28ef380" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953287    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4ca717-cdfc-46cc-93e7-a8bddb3f07d1" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953293    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b2866f7-5408-4815-b3b3-c898ce73181f" containerName="payload-pre-processing"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953298    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d8eef0-e5c5-4f87-9502-973ef001a54e" containerName="payload-processing"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953305    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bb8f64-ee03-4754-89ed-5c69010f4eba" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953311    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac65f6a-7beb-4c50-a0ba-546af2072612" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953317    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca678bf-105c-472c-90e8-3226bdde7e25" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953323    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="500bb411-3bf5-4e1a-bc15-36d332b0af13" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953328    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd73835-f5e2-47b8-8403-3aa7f03492eb" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953339    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a5aa95-faba-4af6-a80e-137745490e73" containerName="cleanup"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953346    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8445b211-0a2f-484d-bfe8-caf789fd035b" containerName="maas-api"
Oct 30 21:30:47.953489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.953352    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48ab868-05de-46c7-89ba-14468f8d778b" containerName="main"
Oct 30 21:30:47.958457 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.958406    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:47.963093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:47.963067    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"]
Oct 30 21:30:48.014484 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.014459    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:30:48.014725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.014707    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:30:48.083822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.083791    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:48.084023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.084004    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:48.084079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.084068    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jndq\" (UniqueName: \"kubernetes.io/projected/0667c9da-083b-4a4d-9178-a6ce96f55939-kube-api-access-2jndq\") pod \"maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:48.185720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.185582    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:48.185966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.185801    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jndq\" (UniqueName: \"kubernetes.io/projected/0667c9da-083b-4a4d-9178-a6ce96f55939-kube-api-access-2jndq\") pod \"maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:48.185966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.185950    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:48.187046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.187020    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:48.193869 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.193829    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:48.202212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.200475    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jndq\" (UniqueName: \"kubernetes.io/projected/0667c9da-083b-4a4d-9178-a6ce96f55939-kube-api-access-2jndq\") pod \"maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:48.294557 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.294461    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:48.590966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.590888    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"]
Oct 30 21:30:48.594950 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:48.594917    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0667c9da_083b_4a4d_9178_a6ce96f55939.slice/crio-fb5fbd0c2d86a19551f604de60984fd3573b38a10a3b7d05e51a4bd078a0dae3 WatchSource:0}: Error finding container fb5fbd0c2d86a19551f604de60984fd3573b38a10a3b7d05e51a4bd078a0dae3: Status 404 returned error can't find the container with id fb5fbd0c2d86a19551f604de60984fd3573b38a10a3b7d05e51a4bd078a0dae3
Oct 30 21:30:48.720625 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:48.720584    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8" event={"ID":"0667c9da-083b-4a4d-9178-a6ce96f55939","Type":"ContainerStarted","Data":"fb5fbd0c2d86a19551f604de60984fd3573b38a10a3b7d05e51a4bd078a0dae3"}
Oct 30 21:30:49.070276 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.070234    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"]
Oct 30 21:30:49.075874 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.075840    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.082212 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.082181    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"]
Oct 30 21:30:49.203338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.203305    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.203511 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.203375    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.203577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.203557    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psdz2\" (UniqueName: \"kubernetes.io/projected/a5751905-174a-4bb2-a85b-f1a6e129a03b-kube-api-access-psdz2\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.305236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.305136    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psdz2\" (UniqueName: \"kubernetes.io/projected/a5751905-174a-4bb2-a85b-f1a6e129a03b-kube-api-access-psdz2\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.305422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.305248    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.305422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.305306    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.306109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.306077    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.308051 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.308024    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.314527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.314502    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psdz2\" (UniqueName: \"kubernetes.io/projected/a5751905-174a-4bb2-a85b-f1a6e129a03b-kube-api-access-psdz2\") pod \"maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.400770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.400741    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:49.684156 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:49.684124    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5751905_174a_4bb2_a85b_f1a6e129a03b.slice/crio-e986720b3b5c3ff9901fe62ef13bd9ac7b6eb71eb880e44e93111b91b33e6c49 WatchSource:0}: Error finding container e986720b3b5c3ff9901fe62ef13bd9ac7b6eb71eb880e44e93111b91b33e6c49: Status 404 returned error can't find the container with id e986720b3b5c3ff9901fe62ef13bd9ac7b6eb71eb880e44e93111b91b33e6c49
Oct 30 21:30:49.684325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.684302    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"]
Oct 30 21:30:49.730525 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.730494    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp" event={"ID":"a5751905-174a-4bb2-a85b-f1a6e129a03b","Type":"ContainerStarted","Data":"e986720b3b5c3ff9901fe62ef13bd9ac7b6eb71eb880e44e93111b91b33e6c49"}
Oct 30 21:30:49.732190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.732165    4810 generic.go:356] "Generic (PLEG): container finished" podID="0667c9da-083b-4a4d-9178-a6ce96f55939" containerID="74adf0ef267c5a63e19af4a03624ba96236756b42fb3a472d285b7125a77493b" exitCode=0
Oct 30 21:30:49.732316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.732243    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8" event={"ID":"0667c9da-083b-4a4d-9178-a6ce96f55939","Type":"ContainerDied","Data":"74adf0ef267c5a63e19af4a03624ba96236756b42fb3a472d285b7125a77493b"}
Oct 30 21:30:49.766158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:49.766122    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d" podUID="3a804f13-7cd2-4949-bd3d-ecf594465e0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:50.743119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:50.743086    4810 generic.go:356] "Generic (PLEG): container finished" podID="a5751905-174a-4bb2-a85b-f1a6e129a03b" containerID="d14da0cf3f9f12c8b12d7b36cfb636c3ae4e2913916d0291a05b3d788c3026c4" exitCode=0
Oct 30 21:30:50.743570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:50.743149    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp" event={"ID":"a5751905-174a-4bb2-a85b-f1a6e129a03b","Type":"ContainerDied","Data":"d14da0cf3f9f12c8b12d7b36cfb636c3ae4e2913916d0291a05b3d788c3026c4"}
Oct 30 21:30:50.972046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:50.971957    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:51.133689 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.133654    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0667c9da-083b-4a4d-9178-a6ce96f55939-kube-api-access-2jndq\" (UniqueName: \"kubernetes.io/projected/0667c9da-083b-4a4d-9178-a6ce96f55939-kube-api-access-2jndq\") pod \"0667c9da-083b-4a4d-9178-a6ce96f55939\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") "
Oct 30 21:30:51.133918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.133743    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-openshift-ca\") pod \"0667c9da-083b-4a4d-9178-a6ce96f55939\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") "
Oct 30 21:30:51.133918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.133835    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-xks-ca\") pod \"0667c9da-083b-4a4d-9178-a6ce96f55939\" (UID: \"0667c9da-083b-4a4d-9178-a6ce96f55939\") "
Oct 30 21:30:51.134218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.134189    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-openshift-ca" pod "0667c9da-083b-4a4d-9178-a6ce96f55939" (UID: "0667c9da-083b-4a4d-9178-a6ce96f55939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:51.134402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.134379    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:51.136156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.136127    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-xks-ca" pod "0667c9da-083b-4a4d-9178-a6ce96f55939" (UID: "0667c9da-083b-4a4d-9178-a6ce96f55939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:51.136156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.136145    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0667c9da-083b-4a4d-9178-a6ce96f55939-kube-api-access-2jndq" pod "0667c9da-083b-4a4d-9178-a6ce96f55939" (UID: "0667c9da-083b-4a4d-9178-a6ce96f55939"). InnerVolumeSpecName "kube-api-access-2jndq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:51.236634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.236180    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0667c9da-083b-4a4d-9178-a6ce96f55939-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:51.236634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.236216    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jndq\" (UniqueName: \"kubernetes.io/projected/0667c9da-083b-4a4d-9178-a6ce96f55939-kube-api-access-2jndq\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:51.250644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.250613    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"]
Oct 30 21:30:51.250919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.250871    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy" containerID="cri-o://ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d" gracePeriod=30
Oct 30 21:30:51.753681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.753651    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"
Oct 30 21:30:51.754124 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.753650    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8" event={"ID":"0667c9da-083b-4a4d-9178-a6ce96f55939","Type":"ContainerDied","Data":"fb5fbd0c2d86a19551f604de60984fd3573b38a10a3b7d05e51a4bd078a0dae3"}
Oct 30 21:30:51.754124 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.753795    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb5fbd0c2d86a19551f604de60984fd3573b38a10a3b7d05e51a4bd078a0dae3"
Oct 30 21:30:51.768271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.768241    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"]
Oct 30 21:30:51.769082 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.769054    4810 cpu_manager.go:414] "containerMap: removing container" podUID="0667c9da-083b-4a4d-9178-a6ce96f55939" containerName="revoke-keys"
Oct 30 21:30:51.769082 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.769078    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0667c9da-083b-4a4d-9178-a6ce96f55939" containerName="revoke-keys"
Oct 30 21:30:51.769350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.769333    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="0667c9da-083b-4a4d-9178-a6ce96f55939" containerName="revoke-keys"
Oct 30 21:30:51.773448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.773430    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:51.784062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.784034    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"]
Oct 30 21:30:51.946227 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.946132    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rdw2\" (UniqueName: \"kubernetes.io/projected/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-kube-api-access-4rdw2\") pod \"maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:51.946375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.946246    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:51.946431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.946399    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:51.949090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.949060    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk" podUID="95743c31-995c-4ff4-b66c-174478a72987" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:51.987652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:51.987626    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:52.033282 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.033241    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"]
Oct 30 21:30:52.038339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.038308    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06d3331e-tw97l"]
Oct 30 21:30:52.047429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.047397    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rdw2\" (UniqueName: \"kubernetes.io/projected/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-kube-api-access-4rdw2\") pod \"maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:52.047581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.047465    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:52.047652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.047594    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:52.048506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.048474    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:52.057287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.057262    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rdw2\" (UniqueName: \"kubernetes.io/projected/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-kube-api-access-4rdw2\") pod \"maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:52.060331 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.060310    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:52.098279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.098244    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:52.149248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.149209    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-xks-ca\") pod \"a5751905-174a-4bb2-a85b-f1a6e129a03b\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") "
Oct 30 21:30:52.149446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.149324    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5751905-174a-4bb2-a85b-f1a6e129a03b-kube-api-access-psdz2\" (UniqueName: \"kubernetes.io/projected/a5751905-174a-4bb2-a85b-f1a6e129a03b-kube-api-access-psdz2\") pod \"a5751905-174a-4bb2-a85b-f1a6e129a03b\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") "
Oct 30 21:30:52.149537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.149516    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-openshift-ca\") pod \"a5751905-174a-4bb2-a85b-f1a6e129a03b\" (UID: \"a5751905-174a-4bb2-a85b-f1a6e129a03b\") "
Oct 30 21:30:52.149882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.149836    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-openshift-ca" pod "a5751905-174a-4bb2-a85b-f1a6e129a03b" (UID: "a5751905-174a-4bb2-a85b-f1a6e129a03b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:52.150095 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.150072    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:52.151949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.151922    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-xks-ca" pod "a5751905-174a-4bb2-a85b-f1a6e129a03b" (UID: "a5751905-174a-4bb2-a85b-f1a6e129a03b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:52.152197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.152171    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5751905-174a-4bb2-a85b-f1a6e129a03b-kube-api-access-psdz2" pod "a5751905-174a-4bb2-a85b-f1a6e129a03b" (UID: "a5751905-174a-4bb2-a85b-f1a6e129a03b"). InnerVolumeSpecName "kube-api-access-psdz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:52.251252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.251218    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5751905-174a-4bb2-a85b-f1a6e129a03b-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:52.251252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.251250    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psdz2\" (UniqueName: \"kubernetes.io/projected/a5751905-174a-4bb2-a85b-f1a6e129a03b-kube-api-access-psdz2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:52.384547 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:30:52.384507    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c8afb2_69b5_41c4_b253_8e61f0ce4010.slice/crio-2a4e12b801650ca5cd6730599e3306e7507e9fb2130222cbed2f6a703e395c52 WatchSource:0}: Error finding container 2a4e12b801650ca5cd6730599e3306e7507e9fb2130222cbed2f6a703e395c52: Status 404 returned error can't find the container with id 2a4e12b801650ca5cd6730599e3306e7507e9fb2130222cbed2f6a703e395c52
Oct 30 21:30:52.385488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.385460    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"]
Oct 30 21:30:52.631605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.631565    4810 patch_prober.go:29] interesting pod/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:30:52.631802 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.631633    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:30:52.782878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.780494    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf" event={"ID":"e9c8afb2-69b5-41c4-b253-8e61f0ce4010","Type":"ContainerStarted","Data":"2a4e12b801650ca5cd6730599e3306e7507e9fb2130222cbed2f6a703e395c52"}
Oct 30 21:30:52.783404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.782886    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"
Oct 30 21:30:52.784028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.783984    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp" event={"ID":"a5751905-174a-4bb2-a85b-f1a6e129a03b","Type":"ContainerDied","Data":"e986720b3b5c3ff9901fe62ef13bd9ac7b6eb71eb880e44e93111b91b33e6c49"}
Oct 30 21:30:52.784169 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.784043    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e986720b3b5c3ff9901fe62ef13bd9ac7b6eb71eb880e44e93111b91b33e6c49"
Oct 30 21:30:52.969533 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:52.969500    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24" podUID="cb523c54-0274-45d9-960d-b395e30205f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:53.066720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.066698    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:30:53.164312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.164235    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a804f13-7cd2-4949-bd3d-ecf594465e0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a804f13-7cd2-4949-bd3d-ecf594465e0b-plugins-config-volume\") pod \"3a804f13-7cd2-4949-bd3d-ecf594465e0b\" (UID: \"3a804f13-7cd2-4949-bd3d-ecf594465e0b\") "
Oct 30 21:30:53.164467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.164451    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a804f13-7cd2-4949-bd3d-ecf594465e0b-kube-api-access-ddr57\" (UniqueName: \"kubernetes.io/projected/3a804f13-7cd2-4949-bd3d-ecf594465e0b-kube-api-access-ddr57\") pod \"3a804f13-7cd2-4949-bd3d-ecf594465e0b\" (UID: \"3a804f13-7cd2-4949-bd3d-ecf594465e0b\") "
Oct 30 21:30:53.164592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.164571    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a804f13-7cd2-4949-bd3d-ecf594465e0b-plugins-config-volume" pod "3a804f13-7cd2-4949-bd3d-ecf594465e0b" (UID: "3a804f13-7cd2-4949-bd3d-ecf594465e0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:53.164871 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.164846    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a804f13-7cd2-4949-bd3d-ecf594465e0b-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:53.166631 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.166606    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a804f13-7cd2-4949-bd3d-ecf594465e0b-kube-api-access-ddr57" pod "3a804f13-7cd2-4949-bd3d-ecf594465e0b" (UID: "3a804f13-7cd2-4949-bd3d-ecf594465e0b"). InnerVolumeSpecName "kube-api-access-ddr57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:53.172158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.172122    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs" podUID="275a0d03-1705-464c-aff5-4b5b8fb668fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:53.172323 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.172296    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz" podUID="029c21db-1fbc-4a68-a513-1fc9da53d370" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:30:53.266398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.266369    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddr57\" (UniqueName: \"kubernetes.io/projected/3a804f13-7cd2-4949-bd3d-ecf594465e0b-kube-api-access-ddr57\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:53.666463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.666421    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aded187d-2e01-42b7-9f54-8279aa364576" path="/var/lib/kubelet/pods/aded187d-2e01-42b7-9f54-8279aa364576/volumes"
Oct 30 21:30:53.797119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.797080    4810 generic.go:356] "Generic (PLEG): container finished" podID="e9c8afb2-69b5-41c4-b253-8e61f0ce4010" containerID="50017bc79ac39b716e7811a63bf452e055c30c0aa37c0b8754d1a982b9f61b1c" exitCode=0
Oct 30 21:30:53.797597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.797164    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf" event={"ID":"e9c8afb2-69b5-41c4-b253-8e61f0ce4010","Type":"ContainerDied","Data":"50017bc79ac39b716e7811a63bf452e055c30c0aa37c0b8754d1a982b9f61b1c"}
Oct 30 21:30:53.799077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.799053    4810 generic.go:356] "Generic (PLEG): container finished" podID="3a804f13-7cd2-4949-bd3d-ecf594465e0b" containerID="f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934" exitCode=0
Oct 30 21:30:53.799242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.799227    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d" event={"ID":"3a804f13-7cd2-4949-bd3d-ecf594465e0b","Type":"ContainerDied","Data":"f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934"}
Oct 30 21:30:53.799312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.799263    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d" event={"ID":"3a804f13-7cd2-4949-bd3d-ecf594465e0b","Type":"ContainerDied","Data":"bd5acda579acf1ef1eb3185ccf868199ad139ed0e3d4ceceecd0658626a8215a"}
Oct 30 21:30:53.799312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.799290    4810 scope.go:122] "RemoveContainer" containerID="f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934"
Oct 30 21:30:53.799424 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.799408    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"
Oct 30 21:30:53.812402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.812384    4810 scope.go:122] "RemoveContainer" containerID="f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934"
Oct 30 21:30:53.812660 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:53.812639    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934\": container with ID starting with f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934 not found: ID does not exist" containerID="f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934"
Oct 30 21:30:53.812750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.812670    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934"} err="failed to get container status \"f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934\": rpc error: code = NotFound desc = could not find container \"f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934\": container with ID starting with f6183fb61c4d3aed6e689a37808de424a10d4a7a21f6dbd5efa21e68095a9934 not found: ID does not exist"
Oct 30 21:30:53.832537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.832500    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"]
Oct 30 21:30:53.836482 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:53.836456    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b78e4-7dc6846c5f-mqx5d"]
Oct 30 21:30:54.972517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:54.972491    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:55.102336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.102252    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-xks-ca\") pod \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") "
Oct 30 21:30:55.102512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.102461    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-kube-api-access-4rdw2\" (UniqueName: \"kubernetes.io/projected/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-kube-api-access-4rdw2\") pod \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") "
Oct 30 21:30:55.102512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.102501    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-openshift-ca\") pod \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\" (UID: \"e9c8afb2-69b5-41c4-b253-8e61f0ce4010\") "
Oct 30 21:30:55.102940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.102912    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-openshift-ca" pod "e9c8afb2-69b5-41c4-b253-8e61f0ce4010" (UID: "e9c8afb2-69b5-41c4-b253-8e61f0ce4010"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:55.103080 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.103066    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:55.104973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.104946    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-kube-api-access-4rdw2" pod "e9c8afb2-69b5-41c4-b253-8e61f0ce4010" (UID: "e9c8afb2-69b5-41c4-b253-8e61f0ce4010"). InnerVolumeSpecName "kube-api-access-4rdw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:55.105105 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.105001    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-xks-ca" pod "e9c8afb2-69b5-41c4-b253-8e61f0ce4010" (UID: "e9c8afb2-69b5-41c4-b253-8e61f0ce4010"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:55.157289 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:55.157251    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:30:55.204289 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.204252    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:55.204289 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.204290    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rdw2\" (UniqueName: \"kubernetes.io/projected/e9c8afb2-69b5-41c4-b253-8e61f0ce4010-kube-api-access-4rdw2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:55.623809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.623776    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:30:55.638785 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.638760    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:30:55.669850 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.669777    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a804f13-7cd2-4949-bd3d-ecf594465e0b" path="/var/lib/kubelet/pods/3a804f13-7cd2-4949-bd3d-ecf594465e0b/volumes"
Oct 30 21:30:55.812307 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.812272    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275a0d03-1705-464c-aff5-4b5b8fb668fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275a0d03-1705-464c-aff5-4b5b8fb668fe-plugins-config-volume\") pod \"275a0d03-1705-464c-aff5-4b5b8fb668fe\" (UID: \"275a0d03-1705-464c-aff5-4b5b8fb668fe\") "
Oct 30 21:30:55.812512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.812396    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029c21db-1fbc-4a68-a513-1fc9da53d370-kube-api-access-tv2vk\" (UniqueName: \"kubernetes.io/projected/029c21db-1fbc-4a68-a513-1fc9da53d370-kube-api-access-tv2vk\") pod \"029c21db-1fbc-4a68-a513-1fc9da53d370\" (UID: \"029c21db-1fbc-4a68-a513-1fc9da53d370\") "
Oct 30 21:30:55.812512 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.812462    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029c21db-1fbc-4a68-a513-1fc9da53d370-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/029c21db-1fbc-4a68-a513-1fc9da53d370-plugins-config-volume\") pod \"029c21db-1fbc-4a68-a513-1fc9da53d370\" (UID: \"029c21db-1fbc-4a68-a513-1fc9da53d370\") "
Oct 30 21:30:55.812672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.812649    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275a0d03-1705-464c-aff5-4b5b8fb668fe-plugins-config-volume" pod "275a0d03-1705-464c-aff5-4b5b8fb668fe" (UID: "275a0d03-1705-464c-aff5-4b5b8fb668fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:55.812743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.812674    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275a0d03-1705-464c-aff5-4b5b8fb668fe-kube-api-access-2d968\" (UniqueName: \"kubernetes.io/projected/275a0d03-1705-464c-aff5-4b5b8fb668fe-kube-api-access-2d968\") pod \"275a0d03-1705-464c-aff5-4b5b8fb668fe\" (UID: \"275a0d03-1705-464c-aff5-4b5b8fb668fe\") "
Oct 30 21:30:55.813011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.812981    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029c21db-1fbc-4a68-a513-1fc9da53d370-plugins-config-volume" pod "029c21db-1fbc-4a68-a513-1fc9da53d370" (UID: "029c21db-1fbc-4a68-a513-1fc9da53d370"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:55.813391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.813368    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/275a0d03-1705-464c-aff5-4b5b8fb668fe-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:55.813483 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.813395    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/029c21db-1fbc-4a68-a513-1fc9da53d370-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:55.815110 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.815078    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275a0d03-1705-464c-aff5-4b5b8fb668fe-kube-api-access-2d968" pod "275a0d03-1705-464c-aff5-4b5b8fb668fe" (UID: "275a0d03-1705-464c-aff5-4b5b8fb668fe"). InnerVolumeSpecName "kube-api-access-2d968". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:55.815444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.815418    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029c21db-1fbc-4a68-a513-1fc9da53d370-kube-api-access-tv2vk" pod "029c21db-1fbc-4a68-a513-1fc9da53d370" (UID: "029c21db-1fbc-4a68-a513-1fc9da53d370"). InnerVolumeSpecName "kube-api-access-tv2vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:55.820521 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.820502    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"
Oct 30 21:30:55.820618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.820520    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf" event={"ID":"e9c8afb2-69b5-41c4-b253-8e61f0ce4010","Type":"ContainerDied","Data":"2a4e12b801650ca5cd6730599e3306e7507e9fb2130222cbed2f6a703e395c52"}
Oct 30 21:30:55.820618 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.820572    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a4e12b801650ca5cd6730599e3306e7507e9fb2130222cbed2f6a703e395c52"
Oct 30 21:30:55.822241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.822220    4810 generic.go:356] "Generic (PLEG): container finished" podID="029c21db-1fbc-4a68-a513-1fc9da53d370" containerID="4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783" exitCode=0
Oct 30 21:30:55.822340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.822270    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"
Oct 30 21:30:55.822340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.822303    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz" event={"ID":"029c21db-1fbc-4a68-a513-1fc9da53d370","Type":"ContainerDied","Data":"4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783"}
Oct 30 21:30:55.822454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.822361    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz" event={"ID":"029c21db-1fbc-4a68-a513-1fc9da53d370","Type":"ContainerDied","Data":"cb82f853598a45971d63b1bfe16ac4d475b47c13de173e3943244168d66a9179"}
Oct 30 21:30:55.822454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.822401    4810 scope.go:122] "RemoveContainer" containerID="4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783"
Oct 30 21:30:55.824400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.824380    4810 generic.go:356] "Generic (PLEG): container finished" podID="275a0d03-1705-464c-aff5-4b5b8fb668fe" containerID="850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953" exitCode=0
Oct 30 21:30:55.824496 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.824425    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"
Oct 30 21:30:55.824496 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.824446    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs" event={"ID":"275a0d03-1705-464c-aff5-4b5b8fb668fe","Type":"ContainerDied","Data":"850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953"}
Oct 30 21:30:55.824587 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.824498    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs" event={"ID":"275a0d03-1705-464c-aff5-4b5b8fb668fe","Type":"ContainerDied","Data":"f9883eed6704509294988eb2b530d3806f773a423b205431ba5a0e08d8c2cd9b"}
Oct 30 21:30:55.834327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.834270    4810 scope.go:122] "RemoveContainer" containerID="4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783"
Oct 30 21:30:55.834592 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:55.834570    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783\": container with ID starting with 4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783 not found: ID does not exist" containerID="4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783"
Oct 30 21:30:55.834668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.834599    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783"} err="failed to get container status \"4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783\": rpc error: code = NotFound desc = could not find container \"4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783\": container with ID starting with 4c2208ce5876c32875ac59c1ad3bf43aedc7c45c634dbe448497a2a4d625f783 not found: ID does not exist"
Oct 30 21:30:55.834668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.834617    4810 scope.go:122] "RemoveContainer" containerID="850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953"
Oct 30 21:30:55.844581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.844558    4810 scope.go:122] "RemoveContainer" containerID="850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953"
Oct 30 21:30:55.844813 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:55.844798    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953\": container with ID starting with 850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953 not found: ID does not exist" containerID="850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953"
Oct 30 21:30:55.844886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.844820    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953"} err="failed to get container status \"850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953\": rpc error: code = NotFound desc = could not find container \"850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953\": container with ID starting with 850ac257defc135efd09cfa9ced654c4c8a279e7250ed1505b39622222374953 not found: ID does not exist"
Oct 30 21:30:55.849530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.849511    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"]
Oct 30 21:30:55.855883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.855852    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a3fb70-6db779f887-zqwtz"]
Oct 30 21:30:55.870705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.870652    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"]
Oct 30 21:30:55.873984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.873961    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a3fb70-64468497c7-9t2bs"]
Oct 30 21:30:55.914844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.914813    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d968\" (UniqueName: \"kubernetes.io/projected/275a0d03-1705-464c-aff5-4b5b8fb668fe-kube-api-access-2d968\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:55.914974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:55.914845    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv2vk\" (UniqueName: \"kubernetes.io/projected/029c21db-1fbc-4a68-a513-1fc9da53d370-kube-api-access-tv2vk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.544364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.544341    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:30:56.623215 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.623173    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-kube-api-access-vmwjx\" (UniqueName: \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-kube-api-access-vmwjx\") pod \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") "
Oct 30 21:30:56.623417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.623233    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-envoy\") pod \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") "
Oct 30 21:30:56.623417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.623269    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-data\") pod \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") "
Oct 30 21:30:56.623417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.623350    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-token\" (UniqueName: \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-token\") pod \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") "
Oct 30 21:30:56.623616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.623416    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-credential-socket\") pod \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") "
Oct 30 21:30:56.623616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.623470    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-podinfo\") pod \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") "
Oct 30 21:30:56.623616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.623520    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-certs\") pod \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") "
Oct 30 21:30:56.623616 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.623578    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-socket\") pod \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") "
Oct 30 21:30:56.623879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.623651    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istiod-ca-cert\") pod \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\" (UID: \"ec9fbac4-7de9-488e-8181-6b9be2b154dd\") "
Oct 30 21:30:56.624373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.624267    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-data" pod "ec9fbac4-7de9-488e-8181-6b9be2b154dd" (UID: "ec9fbac4-7de9-488e-8181-6b9be2b154dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:56.624373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.624299    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-certs" pod "ec9fbac4-7de9-488e-8181-6b9be2b154dd" (UID: "ec9fbac4-7de9-488e-8181-6b9be2b154dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:56.624847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.624539    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-credential-socket" pod "ec9fbac4-7de9-488e-8181-6b9be2b154dd" (UID: "ec9fbac4-7de9-488e-8181-6b9be2b154dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:56.624847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.624618    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-socket" pod "ec9fbac4-7de9-488e-8181-6b9be2b154dd" (UID: "ec9fbac4-7de9-488e-8181-6b9be2b154dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:56.624847 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.624653    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istiod-ca-cert" pod "ec9fbac4-7de9-488e-8181-6b9be2b154dd" (UID: "ec9fbac4-7de9-488e-8181-6b9be2b154dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:56.626663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.626634    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-kube-api-access-vmwjx" pod "ec9fbac4-7de9-488e-8181-6b9be2b154dd" (UID: "ec9fbac4-7de9-488e-8181-6b9be2b154dd"). InnerVolumeSpecName "kube-api-access-vmwjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:56.626803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.626746    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-podinfo" pod "ec9fbac4-7de9-488e-8181-6b9be2b154dd" (UID: "ec9fbac4-7de9-488e-8181-6b9be2b154dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:30:56.626803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.626756    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-envoy" pod "ec9fbac4-7de9-488e-8181-6b9be2b154dd" (UID: "ec9fbac4-7de9-488e-8181-6b9be2b154dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:56.627089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.627066    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-token" pod "ec9fbac4-7de9-488e-8181-6b9be2b154dd" (UID: "ec9fbac4-7de9-488e-8181-6b9be2b154dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:56.725454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.725420    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.725454 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.725453    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.725676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.725464    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.725676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.725473    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.725676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.725482    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.725676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.725491    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmwjx\" (UniqueName: \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-kube-api-access-vmwjx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.725676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.725500    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.725676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.725508    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.725676 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.725516    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9fbac4-7de9-488e-8181-6b9be2b154dd-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.835537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.835456    4810 generic.go:356] "Generic (PLEG): container finished" podID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerID="ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d" exitCode=0
Oct 30 21:30:56.835537 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.835515    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"
Oct 30 21:30:56.835758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.835535    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" event={"ID":"ec9fbac4-7de9-488e-8181-6b9be2b154dd","Type":"ContainerDied","Data":"ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d"}
Oct 30 21:30:56.835758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.835594    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr" event={"ID":"ec9fbac4-7de9-488e-8181-6b9be2b154dd","Type":"ContainerDied","Data":"bfdef695d829dbdc2f199b680f220761b4595046e367b9a6668e1be3d9d1d2cd"}
Oct 30 21:30:56.835758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.835624    4810 scope.go:122] "RemoveContainer" containerID="ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d"
Oct 30 21:30:56.847062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.846992    4810 scope.go:122] "RemoveContainer" containerID="378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732"
Oct 30 21:30:56.857683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.857640    4810 scope.go:122] "RemoveContainer" containerID="ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d"
Oct 30 21:30:56.857908 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:56.857888    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d\": container with ID starting with ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d not found: ID does not exist" containerID="ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d"
Oct 30 21:30:56.857986 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.857917    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d"} err="failed to get container status \"ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d\": rpc error: code = NotFound desc = could not find container \"ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d\": container with ID starting with ba6ffc0ce6a848993ca0bf0fa8e601965c0f146a9e9be703861c163fc0be2d9d not found: ID does not exist"
Oct 30 21:30:56.857986 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.857933    4810 scope.go:122] "RemoveContainer" containerID="378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732"
Oct 30 21:30:56.858200 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:30:56.858183    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732\": container with ID starting with 378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732 not found: ID does not exist" containerID="378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732"
Oct 30 21:30:56.858246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.858205    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732"} err="failed to get container status \"378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732\": rpc error: code = NotFound desc = could not find container \"378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732\": container with ID starting with 378e913adf160bdfb8324a49d03b7336ebb84abe69c899cab14d3de3d4ed8732 not found: ID does not exist"
Oct 30 21:30:56.860571 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.860551    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"]
Oct 30 21:30:56.866719 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:56.866699    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ddcf93-openshift-default-78f6598b5c-v89jr"]
Oct 30 21:30:57.663243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:57.663211    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029c21db-1fbc-4a68-a513-1fc9da53d370" path="/var/lib/kubelet/pods/029c21db-1fbc-4a68-a513-1fc9da53d370/volumes"
Oct 30 21:30:57.663638 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:57.663624    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275a0d03-1705-464c-aff5-4b5b8fb668fe" path="/var/lib/kubelet/pods/275a0d03-1705-464c-aff5-4b5b8fb668fe/volumes"
Oct 30 21:30:57.664093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:57.664079    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" path="/var/lib/kubelet/pods/ec9fbac4-7de9-488e-8181-6b9be2b154dd/volumes"
Oct 30 21:30:58.161527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:58.161493    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"]
Oct 30 21:30:58.183121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:58.181849    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"]
Oct 30 21:30:59.224012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:59.223975    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"]
Oct 30 21:30:59.239948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:30:59.239918    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"]
Oct 30 21:31:01.751127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:01.751098    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"]
Oct 30 21:31:01.755183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:01.755156    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e8063c-j9tnp"]
Oct 30 21:31:01.938074 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:01.938036    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"]
Oct 30 21:31:01.949333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:01.949297    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk" podUID="95743c31-995c-4ff4-b66c-174478a72987" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:01.978648 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:01.978609    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"]
Oct 30 21:31:02.968588 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:02.968549    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24" podUID="cb523c54-0274-45d9-960d-b395e30205f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:03.162554 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:03.162488    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks" podUID="e6089936-7fbd-42fa-b030-20e1af0a8e2b" containerName="payload-processing" containerID="cri-o://00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789" gracePeriod=25
Oct 30 21:31:03.183027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:03.182991    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt" podUID="3fb1e381-15d1-4be5-ae3b-995385a56b10" containerName="payload-pre-processing" containerID="cri-o://1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407" gracePeriod=25
Oct 30 21:31:03.663288 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:03.663256    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5751905-174a-4bb2-a85b-f1a6e129a03b" path="/var/lib/kubelet/pods/a5751905-174a-4bb2-a85b-f1a6e129a03b/volumes"
Oct 30 21:31:04.224279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:04.224230    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6" podUID="967070ba-c779-46ce-8256-cd93fab76c38" containerName="payload-processing" containerID="cri-o://8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2" gracePeriod=25
Oct 30 21:31:04.240753 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:04.240704    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx" podUID="7a8c293f-f822-4033-99cc-0640e9013e20" containerName="payload-pre-processing" containerID="cri-o://2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840" gracePeriod=25
Oct 30 21:31:04.615907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:04.615770    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx" podUID="7a8c293f-f822-4033-99cc-0640e9013e20" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:06.308696 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.308659    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"]
Oct 30 21:31:06.312151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.312125    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-99668e-4n4b8"]
Oct 30 21:31:06.613779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.613749    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:31:06.640389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.640349    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6" podUID="967070ba-c779-46ce-8256-cd93fab76c38" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:06.647390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.647361    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:31:06.720207 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.720175    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb523c54-0274-45d9-960d-b395e30205f1-kube-api-access-8shgb\" (UniqueName: \"kubernetes.io/projected/cb523c54-0274-45d9-960d-b395e30205f1-kube-api-access-8shgb\") pod \"cb523c54-0274-45d9-960d-b395e30205f1\" (UID: \"cb523c54-0274-45d9-960d-b395e30205f1\") "
Oct 30 21:31:06.720412 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.720219    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb523c54-0274-45d9-960d-b395e30205f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb523c54-0274-45d9-960d-b395e30205f1-plugins-config-volume\") pod \"cb523c54-0274-45d9-960d-b395e30205f1\" (UID: \"cb523c54-0274-45d9-960d-b395e30205f1\") "
Oct 30 21:31:06.720679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.720655    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb523c54-0274-45d9-960d-b395e30205f1-plugins-config-volume" pod "cb523c54-0274-45d9-960d-b395e30205f1" (UID: "cb523c54-0274-45d9-960d-b395e30205f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:06.722423 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.722402    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb523c54-0274-45d9-960d-b395e30205f1-kube-api-access-8shgb" pod "cb523c54-0274-45d9-960d-b395e30205f1" (UID: "cb523c54-0274-45d9-960d-b395e30205f1"). InnerVolumeSpecName "kube-api-access-8shgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:06.821359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.821273    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95743c31-995c-4ff4-b66c-174478a72987-kube-api-access-qq65z\" (UniqueName: \"kubernetes.io/projected/95743c31-995c-4ff4-b66c-174478a72987-kube-api-access-qq65z\") pod \"95743c31-995c-4ff4-b66c-174478a72987\" (UID: \"95743c31-995c-4ff4-b66c-174478a72987\") "
Oct 30 21:31:06.821506 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.821492    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95743c31-995c-4ff4-b66c-174478a72987-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95743c31-995c-4ff4-b66c-174478a72987-plugins-config-volume\") pod \"95743c31-995c-4ff4-b66c-174478a72987\" (UID: \"95743c31-995c-4ff4-b66c-174478a72987\") "
Oct 30 21:31:06.822151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.821841    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95743c31-995c-4ff4-b66c-174478a72987-plugins-config-volume" pod "95743c31-995c-4ff4-b66c-174478a72987" (UID: "95743c31-995c-4ff4-b66c-174478a72987"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:06.822151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.822104    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8shgb\" (UniqueName: \"kubernetes.io/projected/cb523c54-0274-45d9-960d-b395e30205f1-kube-api-access-8shgb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:06.822151 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.822126    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb523c54-0274-45d9-960d-b395e30205f1-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:06.822384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.822159    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95743c31-995c-4ff4-b66c-174478a72987-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:06.823459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.823438    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95743c31-995c-4ff4-b66c-174478a72987-kube-api-access-qq65z" pod "95743c31-995c-4ff4-b66c-174478a72987" (UID: "95743c31-995c-4ff4-b66c-174478a72987"). InnerVolumeSpecName "kube-api-access-qq65z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:06.923737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.923696    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq65z\" (UniqueName: \"kubernetes.io/projected/95743c31-995c-4ff4-b66c-174478a72987-kube-api-access-qq65z\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:06.938797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.938723    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj" podUID="f268e880-2e55-4ff3-837d-a4fb2abbf8a1" containerName="payload-processing" containerID="cri-o://d9d0439b665373fee670ec399653e8c1d39db1b98ad5a30a3a2548ece0f5f256" gracePeriod=25
Oct 30 21:31:06.939265 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.939224    4810 generic.go:356] "Generic (PLEG): container finished" podID="95743c31-995c-4ff4-b66c-174478a72987" containerID="40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f" exitCode=0
Oct 30 21:31:06.939400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.939310    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"
Oct 30 21:31:06.939400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.939316    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk" event={"ID":"95743c31-995c-4ff4-b66c-174478a72987","Type":"ContainerDied","Data":"40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f"}
Oct 30 21:31:06.939400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.939386    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk" event={"ID":"95743c31-995c-4ff4-b66c-174478a72987","Type":"ContainerDied","Data":"98144e32d301799b7b08331264fb8615299d176af4f7bf25446a49fc7a554234"}
Oct 30 21:31:06.939611 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.939418    4810 scope.go:122] "RemoveContainer" containerID="40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f"
Oct 30 21:31:06.941403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.941377    4810 generic.go:356] "Generic (PLEG): container finished" podID="cb523c54-0274-45d9-960d-b395e30205f1" containerID="3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a" exitCode=0
Oct 30 21:31:06.941509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.941406    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24" event={"ID":"cb523c54-0274-45d9-960d-b395e30205f1","Type":"ContainerDied","Data":"3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a"}
Oct 30 21:31:06.941509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.941433    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"
Oct 30 21:31:06.943941 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.943907    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24" event={"ID":"cb523c54-0274-45d9-960d-b395e30205f1","Type":"ContainerDied","Data":"07fa9e4dde2e7fe7e66566011ced3cc85ccc593cb6017953ec5333f38d6a6a23"}
Oct 30 21:31:06.953223 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.953203    4810 scope.go:122] "RemoveContainer" containerID="40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f"
Oct 30 21:31:06.953489 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:06.953472    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f\": container with ID starting with 40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f not found: ID does not exist" containerID="40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f"
Oct 30 21:31:06.953540 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.953495    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f"} err="failed to get container status \"40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f\": rpc error: code = NotFound desc = could not find container \"40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f\": container with ID starting with 40e3f1bbfaabce1276fc2e21d69c98668a1739ac743ffc950cf721e7c902c25f not found: ID does not exist"
Oct 30 21:31:06.953540 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.953511    4810 scope.go:122] "RemoveContainer" containerID="3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a"
Oct 30 21:31:06.964842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.964819    4810 scope.go:122] "RemoveContainer" containerID="3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a"
Oct 30 21:31:06.965162 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:06.965135    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a\": container with ID starting with 3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a not found: ID does not exist" containerID="3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a"
Oct 30 21:31:06.965230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.965172    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a"} err="failed to get container status \"3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a\": rpc error: code = NotFound desc = could not find container \"3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a\": container with ID starting with 3a1d10660337a09ae4ee1dd1e4cc99d083a70c14998c8d1bbbb538566a1d969a not found: ID does not exist"
Oct 30 21:31:06.966295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.966278    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"]
Oct 30 21:31:06.970930 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.970906    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddcf93-56fc6ff47f-l6zsk"]
Oct 30 21:31:06.979908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.979837    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2" podUID="a6c727b1-6fe8-46eb-95ff-4748528ee68d" containerName="payload-pre-processing" containerID="cri-o://128afce246826cbda2b3a0ec5b0a922a8f746654ede8d6869d97174b578e0049" gracePeriod=25
Oct 30 21:31:06.984124 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.984103    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"]
Oct 30 21:31:06.989203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:06.989181    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddcf93-75fd559bb7-g7v24"]
Oct 30 21:31:07.107609 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:07.107467    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj" podUID="f268e880-2e55-4ff3-837d-a4fb2abbf8a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:07.107907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:07.107879    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2" podUID="a6c727b1-6fe8-46eb-95ff-4748528ee68d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:07.442661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:07.442624    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"]
Oct 30 21:31:07.448899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:07.446971    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5506f4-f9gvf"]
Oct 30 21:31:07.662740 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:07.662701    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0667c9da-083b-4a4d-9178-a6ce96f55939" path="/var/lib/kubelet/pods/0667c9da-083b-4a4d-9178-a6ce96f55939/volumes"
Oct 30 21:31:07.663188 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:07.663171    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95743c31-995c-4ff4-b66c-174478a72987" path="/var/lib/kubelet/pods/95743c31-995c-4ff4-b66c-174478a72987/volumes"
Oct 30 21:31:07.663550 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:07.663538    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb523c54-0274-45d9-960d-b395e30205f1" path="/var/lib/kubelet/pods/cb523c54-0274-45d9-960d-b395e30205f1/volumes"
Oct 30 21:31:07.664263 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:07.664248    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c8afb2-69b5-41c4-b253-8e61f0ce4010" path="/var/lib/kubelet/pods/e9c8afb2-69b5-41c4-b253-8e61f0ce4010/volumes"
Oct 30 21:31:07.755822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:07.755732    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks" podUID="e6089936-7fbd-42fa-b030-20e1af0a8e2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:07.755997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:07.755830    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt" podUID="3fb1e381-15d1-4be5-ae3b-995385a56b10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:10.509252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.509216    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"]
Oct 30 21:31:10.510797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510767    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy"
Oct 30 21:31:10.510797 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510799    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510820    4810 cpu_manager.go:414] "containerMap: removing container" podUID="95743c31-995c-4ff4-b66c-174478a72987" containerName="payload-pre-processing"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510828    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95743c31-995c-4ff4-b66c-174478a72987" containerName="payload-pre-processing"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510846    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3a804f13-7cd2-4949-bd3d-ecf594465e0b" containerName="payload-pre-processing"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510888    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a804f13-7cd2-4949-bd3d-ecf594465e0b" containerName="payload-pre-processing"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510931    4810 cpu_manager.go:414] "containerMap: removing container" podUID="029c21db-1fbc-4a68-a513-1fc9da53d370" containerName="payload-pre-processing"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510940    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029c21db-1fbc-4a68-a513-1fc9da53d370" containerName="payload-pre-processing"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510960    4810 cpu_manager.go:414] "containerMap: removing container" podUID="275a0d03-1705-464c-aff5-4b5b8fb668fe" containerName="payload-processing"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510968    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275a0d03-1705-464c-aff5-4b5b8fb668fe" containerName="payload-processing"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.510994    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a5751905-174a-4bb2-a85b-f1a6e129a03b" containerName="revoke-keys"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511001    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5751905-174a-4bb2-a85b-f1a6e129a03b" containerName="revoke-keys"
Oct 30 21:31:10.511022 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511024    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cb523c54-0274-45d9-960d-b395e30205f1" containerName="payload-processing"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511031    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb523c54-0274-45d9-960d-b395e30205f1" containerName="payload-processing"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511051    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e9c8afb2-69b5-41c4-b253-8e61f0ce4010" containerName="revoke-keys"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511058    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c8afb2-69b5-41c4-b253-8e61f0ce4010" containerName="revoke-keys"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511219    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="029c21db-1fbc-4a68-a513-1fc9da53d370" containerName="payload-pre-processing"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511236    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511247    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="275a0d03-1705-464c-aff5-4b5b8fb668fe" containerName="payload-processing"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511262    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5751905-174a-4bb2-a85b-f1a6e129a03b" containerName="revoke-keys"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511272    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb523c54-0274-45d9-960d-b395e30205f1" containerName="payload-processing"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511283    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c8afb2-69b5-41c4-b253-8e61f0ce4010" containerName="revoke-keys"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511294    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511305    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="95743c31-995c-4ff4-b66c-174478a72987" containerName="payload-pre-processing"
Oct 30 21:31:10.511408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.511316    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a804f13-7cd2-4949-bd3d-ecf594465e0b" containerName="payload-pre-processing"
Oct 30 21:31:10.522070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.522037    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"]
Oct 30 21:31:10.522192 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.522160    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:10.529886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.526707    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:10.529886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.526792    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:31:10.676052 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.676020    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:10.676202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.676108    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:10.676202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.676151    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9jhq\" (UniqueName: \"kubernetes.io/projected/a17faf1b-080e-4e34-8c63-e371f0dbe493-kube-api-access-n9jhq\") pod \"maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:10.778061 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.777981    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:10.778061 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.778053    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9jhq\" (UniqueName: \"kubernetes.io/projected/a17faf1b-080e-4e34-8c63-e371f0dbe493-kube-api-access-n9jhq\") pod \"maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:10.778271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.778227    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:10.779007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.778971    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:10.780699 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.780668    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:10.787190 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.787159    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9jhq\" (UniqueName: \"kubernetes.io/projected/a17faf1b-080e-4e34-8c63-e371f0dbe493-kube-api-access-n9jhq\") pod \"maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:10.839088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:10.839065    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:11.126264 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:11.126236    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"]
Oct 30 21:31:11.127337 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:31:11.127309    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17faf1b_080e_4e34_8c63_e371f0dbe493.slice/crio-196f2a4c400a1e4b5c56a859160d372ec771fcd584c5818caced8c9156395d1f WatchSource:0}: Error finding container 196f2a4c400a1e4b5c56a859160d372ec771fcd584c5818caced8c9156395d1f: Status 404 returned error can't find the container with id 196f2a4c400a1e4b5c56a859160d372ec771fcd584c5818caced8c9156395d1f
Oct 30 21:31:12.006251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:12.006218    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr" event={"ID":"a17faf1b-080e-4e34-8c63-e371f0dbe493","Type":"ContainerStarted","Data":"4c7e7b03f631ac9a15d763c00e5d4236fe2df7937ef98e1c38d73c665db400fe"}
Oct 30 21:31:12.006751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:12.006277    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr" event={"ID":"a17faf1b-080e-4e34-8c63-e371f0dbe493","Type":"ContainerStarted","Data":"196f2a4c400a1e4b5c56a859160d372ec771fcd584c5818caced8c9156395d1f"}
Oct 30 21:31:12.022832 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:12.022783    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr" podStartSLOduration=2.022770524 podStartE2EDuration="2.022770524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:12.022396931 +0000 UTC m=+3238.514213471" watchObservedRunningTime="2026-10-30 21:31:12.022770524 +0000 UTC m=+3238.514587068"
Oct 30 21:31:13.017636 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:13.017597    4810 generic.go:356] "Generic (PLEG): container finished" podID="a17faf1b-080e-4e34-8c63-e371f0dbe493" containerID="4c7e7b03f631ac9a15d763c00e5d4236fe2df7937ef98e1c38d73c665db400fe" exitCode=0
Oct 30 21:31:13.018069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:13.017686    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr" event={"ID":"a17faf1b-080e-4e34-8c63-e371f0dbe493","Type":"ContainerDied","Data":"4c7e7b03f631ac9a15d763c00e5d4236fe2df7937ef98e1c38d73c665db400fe"}
Oct 30 21:31:14.230966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.230936    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:14.240844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.240818    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-xks-ca\") pod \"a17faf1b-080e-4e34-8c63-e371f0dbe493\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") "
Oct 30 21:31:14.240970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.240879    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17faf1b-080e-4e34-8c63-e371f0dbe493-kube-api-access-n9jhq\" (UniqueName: \"kubernetes.io/projected/a17faf1b-080e-4e34-8c63-e371f0dbe493-kube-api-access-n9jhq\") pod \"a17faf1b-080e-4e34-8c63-e371f0dbe493\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") "
Oct 30 21:31:14.241029 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.241016    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-openshift-ca\") pod \"a17faf1b-080e-4e34-8c63-e371f0dbe493\" (UID: \"a17faf1b-080e-4e34-8c63-e371f0dbe493\") "
Oct 30 21:31:14.241419 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.241382    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-openshift-ca" pod "a17faf1b-080e-4e34-8c63-e371f0dbe493" (UID: "a17faf1b-080e-4e34-8c63-e371f0dbe493"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:14.241598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.241583    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:14.243025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.243003    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17faf1b-080e-4e34-8c63-e371f0dbe493-kube-api-access-n9jhq" pod "a17faf1b-080e-4e34-8c63-e371f0dbe493" (UID: "a17faf1b-080e-4e34-8c63-e371f0dbe493"). InnerVolumeSpecName "kube-api-access-n9jhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:14.243025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.243011    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-xks-ca" pod "a17faf1b-080e-4e34-8c63-e371f0dbe493" (UID: "a17faf1b-080e-4e34-8c63-e371f0dbe493"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:14.345957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.345849    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a17faf1b-080e-4e34-8c63-e371f0dbe493-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:14.346245 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.346214    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9jhq\" (UniqueName: \"kubernetes.io/projected/a17faf1b-080e-4e34-8c63-e371f0dbe493-kube-api-access-n9jhq\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:14.351712 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.351678    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"]
Oct 30 21:31:14.357183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.357156    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a17faf1b-080e-4e34-8c63-e371f0dbe493" containerName="revoke-keys"
Oct 30 21:31:14.363642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.360610    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17faf1b-080e-4e34-8c63-e371f0dbe493" containerName="revoke-keys"
Oct 30 21:31:14.363642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.360713    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy"
Oct 30 21:31:14.363642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.360724    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9fbac4-7de9-488e-8181-6b9be2b154dd" containerName="istio-proxy"
Oct 30 21:31:14.363642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.360983    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17faf1b-080e-4e34-8c63-e371f0dbe493" containerName="revoke-keys"
Oct 30 21:31:14.366077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.365934    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"]
Oct 30 21:31:14.366238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.366035    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:14.447169 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.447135    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzq6\" (UniqueName: \"kubernetes.io/projected/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-kube-api-access-zfzq6\") pod \"maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:14.447317 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.447253    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:14.447317 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.447284    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:14.548332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.548303    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzq6\" (UniqueName: \"kubernetes.io/projected/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-kube-api-access-zfzq6\") pod \"maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:14.548489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.548351    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:14.548489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.548376    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:14.549077 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.549054    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:14.551337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.551313    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:14.571405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.571376    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzq6\" (UniqueName: \"kubernetes.io/projected/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-kube-api-access-zfzq6\") pod \"maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:14.616224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.616144    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx" podUID="7a8c293f-f822-4033-99cc-0640e9013e20" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:14.685515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:14.685485    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:15.028771 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:31:15.028738    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a5d622_b35e_4899_a5d4_a37a0bcff6dd.slice/crio-6f11e98190cd455406980f802300f15fbd27ff006b0634e1d761f1abe6ee965a WatchSource:0}: Error finding container 6f11e98190cd455406980f802300f15fbd27ff006b0634e1d761f1abe6ee965a: Status 404 returned error can't find the container with id 6f11e98190cd455406980f802300f15fbd27ff006b0634e1d761f1abe6ee965a
Oct 30 21:31:15.030318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.030286    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"]
Oct 30 21:31:15.054925 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.054898    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n" event={"ID":"19a5d622-b35e-4899-a5d4-a37a0bcff6dd","Type":"ContainerStarted","Data":"6f11e98190cd455406980f802300f15fbd27ff006b0634e1d761f1abe6ee965a"}
Oct 30 21:31:15.056510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.056488    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"
Oct 30 21:31:15.056620 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.056524    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr" event={"ID":"a17faf1b-080e-4e34-8c63-e371f0dbe493","Type":"ContainerDied","Data":"196f2a4c400a1e4b5c56a859160d372ec771fcd584c5818caced8c9156395d1f"}
Oct 30 21:31:15.056620 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.056579    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="196f2a4c400a1e4b5c56a859160d372ec771fcd584c5818caced8c9156395d1f"
Oct 30 21:31:15.164884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.164834    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"]
Oct 30 21:31:15.170987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.170965    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.179645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.179614    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"]
Oct 30 21:31:15.266978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.266949    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.267342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.266985    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.267342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.267028    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlj6s\" (UniqueName: \"kubernetes.io/projected/d9b133d7-be5b-4088-80aa-648b7c4c59ad-kube-api-access-wlj6s\") pod \"maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.369083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.369036    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.369271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.369089    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.369271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.369146    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlj6s\" (UniqueName: \"kubernetes.io/projected/d9b133d7-be5b-4088-80aa-648b7c4c59ad-kube-api-access-wlj6s\") pod \"maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.370202 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.370165    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.372139 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.372095    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.396369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.396324    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlj6s\" (UniqueName: \"kubernetes.io/projected/d9b133d7-be5b-4088-80aa-648b7c4c59ad-kube-api-access-wlj6s\") pod \"maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.596644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.596555    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:15.898298 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:31:15.898267    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b133d7_be5b_4088_80aa_648b7c4c59ad.slice/crio-98061348a01cc7feaed7e6c5ceb4309a88b91c56a7a35130cdce6af77e61451f WatchSource:0}: Error finding container 98061348a01cc7feaed7e6c5ceb4309a88b91c56a7a35130cdce6af77e61451f: Status 404 returned error can't find the container with id 98061348a01cc7feaed7e6c5ceb4309a88b91c56a7a35130cdce6af77e61451f
Oct 30 21:31:15.898938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:15.898908    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"]
Oct 30 21:31:16.067063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:16.067028    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n" event={"ID":"19a5d622-b35e-4899-a5d4-a37a0bcff6dd","Type":"ContainerStarted","Data":"3e5d391c8bb3c3d4c3356fe6249f8f89ebc339258b74cd0296778f8ac2e7416b"}
Oct 30 21:31:16.068653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:16.068616    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh" event={"ID":"d9b133d7-be5b-4088-80aa-648b7c4c59ad","Type":"ContainerStarted","Data":"98061348a01cc7feaed7e6c5ceb4309a88b91c56a7a35130cdce6af77e61451f"}
Oct 30 21:31:16.103097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:16.103054    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n" podStartSLOduration=2.103039609 podStartE2EDuration="2.103039609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:16.100340256 +0000 UTC m=+3242.592156803" watchObservedRunningTime="2026-10-30 21:31:16.103039609 +0000 UTC m=+3242.594856167"
Oct 30 21:31:16.635909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:16.635853    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6" podUID="967070ba-c779-46ce-8256-cd93fab76c38" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:17.101257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:17.101163    4810 generic.go:356] "Generic (PLEG): container finished" podID="19a5d622-b35e-4899-a5d4-a37a0bcff6dd" containerID="3e5d391c8bb3c3d4c3356fe6249f8f89ebc339258b74cd0296778f8ac2e7416b" exitCode=0
Oct 30 21:31:17.101536 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:17.101497    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n" event={"ID":"19a5d622-b35e-4899-a5d4-a37a0bcff6dd","Type":"ContainerDied","Data":"3e5d391c8bb3c3d4c3356fe6249f8f89ebc339258b74cd0296778f8ac2e7416b"}
Oct 30 21:31:17.103513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:17.103488    4810 generic.go:356] "Generic (PLEG): container finished" podID="d9b133d7-be5b-4088-80aa-648b7c4c59ad" containerID="eb37682d0b71be97e592aec7b2250f33255d8e9dfde343359a5ce01187ae5f66" exitCode=0
Oct 30 21:31:17.103633 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:17.103551    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh" event={"ID":"d9b133d7-be5b-4088-80aa-648b7c4c59ad","Type":"ContainerDied","Data":"eb37682d0b71be97e592aec7b2250f33255d8e9dfde343359a5ce01187ae5f66"}
Oct 30 21:31:17.106130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:17.106102    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2" podUID="a6c727b1-6fe8-46eb-95ff-4748528ee68d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:17.106239 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:17.106102    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj" podUID="f268e880-2e55-4ff3-837d-a4fb2abbf8a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:17.756236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:17.756185    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt" podUID="3fb1e381-15d1-4be5-ae3b-995385a56b10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:17.756236 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:17.756195    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks" podUID="e6089936-7fbd-42fa-b030-20e1af0a8e2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:18.316195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.316161    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:18.320275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.320256    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:18.421164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.421131    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-xks-ca\") pod \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") "
Oct 30 21:31:18.421369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.421174    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-xks-ca\") pod \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") "
Oct 30 21:31:18.421369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.421275    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-kube-api-access-zfzq6\" (UniqueName: \"kubernetes.io/projected/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-kube-api-access-zfzq6\") pod \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") "
Oct 30 21:31:18.421369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.421300    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-openshift-ca\") pod \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") "
Oct 30 21:31:18.421574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.421407    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-openshift-ca\") pod \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\" (UID: \"19a5d622-b35e-4899-a5d4-a37a0bcff6dd\") "
Oct 30 21:31:18.421574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.421442    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b133d7-be5b-4088-80aa-648b7c4c59ad-kube-api-access-wlj6s\" (UniqueName: \"kubernetes.io/projected/d9b133d7-be5b-4088-80aa-648b7c4c59ad-kube-api-access-wlj6s\") pod \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\" (UID: \"d9b133d7-be5b-4088-80aa-648b7c4c59ad\") "
Oct 30 21:31:18.421827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.421764    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-openshift-ca" pod "d9b133d7-be5b-4088-80aa-648b7c4c59ad" (UID: "d9b133d7-be5b-4088-80aa-648b7c4c59ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:18.422034 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.422004    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:18.422174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.422147    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-openshift-ca" pod "19a5d622-b35e-4899-a5d4-a37a0bcff6dd" (UID: "19a5d622-b35e-4899-a5d4-a37a0bcff6dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:18.423429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.423408    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-xks-ca" pod "d9b133d7-be5b-4088-80aa-648b7c4c59ad" (UID: "d9b133d7-be5b-4088-80aa-648b7c4c59ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:18.423796 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.423770    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-kube-api-access-zfzq6" pod "19a5d622-b35e-4899-a5d4-a37a0bcff6dd" (UID: "19a5d622-b35e-4899-a5d4-a37a0bcff6dd"). InnerVolumeSpecName "kube-api-access-zfzq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:18.423897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.423841    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-xks-ca" pod "19a5d622-b35e-4899-a5d4-a37a0bcff6dd" (UID: "19a5d622-b35e-4899-a5d4-a37a0bcff6dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:18.423970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.423899    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b133d7-be5b-4088-80aa-648b7c4c59ad-kube-api-access-wlj6s" pod "d9b133d7-be5b-4088-80aa-648b7c4c59ad" (UID: "d9b133d7-be5b-4088-80aa-648b7c4c59ad"). InnerVolumeSpecName "kube-api-access-wlj6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:18.523590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.523558    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:18.523590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.523588    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlj6s\" (UniqueName: \"kubernetes.io/projected/d9b133d7-be5b-4088-80aa-648b7c4c59ad-kube-api-access-wlj6s\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:18.523789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.523598    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b133d7-be5b-4088-80aa-648b7c4c59ad-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:18.523789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.523608    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:18.523789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:18.523618    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfzq6\" (UniqueName: \"kubernetes.io/projected/19a5d622-b35e-4899-a5d4-a37a0bcff6dd-kube-api-access-zfzq6\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:19.122055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:19.122024    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"
Oct 30 21:31:19.122480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:19.122056    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh" event={"ID":"d9b133d7-be5b-4088-80aa-648b7c4c59ad","Type":"ContainerDied","Data":"98061348a01cc7feaed7e6c5ceb4309a88b91c56a7a35130cdce6af77e61451f"}
Oct 30 21:31:19.122480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:19.122107    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98061348a01cc7feaed7e6c5ceb4309a88b91c56a7a35130cdce6af77e61451f"
Oct 30 21:31:19.123770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:19.123750    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"
Oct 30 21:31:19.123896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:19.123797    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n" event={"ID":"19a5d622-b35e-4899-a5d4-a37a0bcff6dd","Type":"ContainerDied","Data":"6f11e98190cd455406980f802300f15fbd27ff006b0634e1d761f1abe6ee965a"}
Oct 30 21:31:19.123896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:19.123846    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f11e98190cd455406980f802300f15fbd27ff006b0634e1d761f1abe6ee965a"
Oct 30 21:31:19.376023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:19.375944    4810 scope.go:122] "RemoveContainer" containerID="cc2fabf765d0c606dd6432aa9dbe20d52f8bc39bc817b3594bf80cae9e9e7446"
Oct 30 21:31:20.659799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:20.659763    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"]
Oct 30 21:31:20.678100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:20.676255    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"]
Oct 30 21:31:23.524288 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.524263    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:31:23.528564 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.528544    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:31:23.593578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.593527    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6089936-7fbd-42fa-b030-20e1af0a8e2b-kube-api-access-zx4f8\" (UniqueName: \"kubernetes.io/projected/e6089936-7fbd-42fa-b030-20e1af0a8e2b-kube-api-access-zx4f8\") pod \"e6089936-7fbd-42fa-b030-20e1af0a8e2b\" (UID: \"e6089936-7fbd-42fa-b030-20e1af0a8e2b\") "
Oct 30 21:31:23.593764 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.593695    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6089936-7fbd-42fa-b030-20e1af0a8e2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6089936-7fbd-42fa-b030-20e1af0a8e2b-plugins-config-volume\") pod \"e6089936-7fbd-42fa-b030-20e1af0a8e2b\" (UID: \"e6089936-7fbd-42fa-b030-20e1af0a8e2b\") "
Oct 30 21:31:23.593917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.593883    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb1e381-15d1-4be5-ae3b-995385a56b10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb1e381-15d1-4be5-ae3b-995385a56b10-plugins-config-volume\") pod \"3fb1e381-15d1-4be5-ae3b-995385a56b10\" (UID: \"3fb1e381-15d1-4be5-ae3b-995385a56b10\") "
Oct 30 21:31:23.594050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.594032    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb1e381-15d1-4be5-ae3b-995385a56b10-kube-api-access-2gg64\" (UniqueName: \"kubernetes.io/projected/3fb1e381-15d1-4be5-ae3b-995385a56b10-kube-api-access-2gg64\") pod \"3fb1e381-15d1-4be5-ae3b-995385a56b10\" (UID: \"3fb1e381-15d1-4be5-ae3b-995385a56b10\") "
Oct 30 21:31:23.595334 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.595237    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb1e381-15d1-4be5-ae3b-995385a56b10-plugins-config-volume" pod "3fb1e381-15d1-4be5-ae3b-995385a56b10" (UID: "3fb1e381-15d1-4be5-ae3b-995385a56b10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:23.595998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.595244    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6089936-7fbd-42fa-b030-20e1af0a8e2b-plugins-config-volume" pod "e6089936-7fbd-42fa-b030-20e1af0a8e2b" (UID: "e6089936-7fbd-42fa-b030-20e1af0a8e2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:23.642903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.635659    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb1e381-15d1-4be5-ae3b-995385a56b10-kube-api-access-2gg64" pod "3fb1e381-15d1-4be5-ae3b-995385a56b10" (UID: "3fb1e381-15d1-4be5-ae3b-995385a56b10"). InnerVolumeSpecName "kube-api-access-2gg64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:23.697177 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.696494    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6089936-7fbd-42fa-b030-20e1af0a8e2b-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:23.697177 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.696533    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb1e381-15d1-4be5-ae3b-995385a56b10-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:23.697177 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.696548    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gg64\" (UniqueName: \"kubernetes.io/projected/3fb1e381-15d1-4be5-ae3b-995385a56b10-kube-api-access-2gg64\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:23.706890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.701354    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6089936-7fbd-42fa-b030-20e1af0a8e2b-kube-api-access-zx4f8" pod "e6089936-7fbd-42fa-b030-20e1af0a8e2b" (UID: "e6089936-7fbd-42fa-b030-20e1af0a8e2b"). InnerVolumeSpecName "kube-api-access-zx4f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:23.790526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.790493    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"]
Oct 30 21:31:23.794487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.794457    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e1e0f5-zqnlr"]
Oct 30 21:31:23.797726 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:23.797700    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx4f8\" (UniqueName: \"kubernetes.io/projected/e6089936-7fbd-42fa-b030-20e1af0a8e2b-kube-api-access-zx4f8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:24.167767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.167732    4810 generic.go:356] "Generic (PLEG): container finished" podID="e6089936-7fbd-42fa-b030-20e1af0a8e2b" containerID="00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789" exitCode=0
Oct 30 21:31:24.168019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.167789    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"
Oct 30 21:31:24.168019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.167816    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks" event={"ID":"e6089936-7fbd-42fa-b030-20e1af0a8e2b","Type":"ContainerDied","Data":"00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789"}
Oct 30 21:31:24.168019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.167900    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks" event={"ID":"e6089936-7fbd-42fa-b030-20e1af0a8e2b","Type":"ContainerDied","Data":"33fb9ebac34cdfc377c262a939e1845d69d3d01df05e498b415340b152f3ccc5"}
Oct 30 21:31:24.168019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.167928    4810 scope.go:122] "RemoveContainer" containerID="00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789"
Oct 30 21:31:24.169675 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.169649    4810 generic.go:356] "Generic (PLEG): container finished" podID="3fb1e381-15d1-4be5-ae3b-995385a56b10" containerID="1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407" exitCode=0
Oct 30 21:31:24.169806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.169694    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"
Oct 30 21:31:24.169806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.169773    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt" event={"ID":"3fb1e381-15d1-4be5-ae3b-995385a56b10","Type":"ContainerDied","Data":"1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407"}
Oct 30 21:31:24.170068 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.169808    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt" event={"ID":"3fb1e381-15d1-4be5-ae3b-995385a56b10","Type":"ContainerDied","Data":"31e2e57c8fbcf7c31c04bfa3474779580c7d7988c18bfc0e46f2a967d4e678aa"}
Oct 30 21:31:24.191296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.191272    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"]
Oct 30 21:31:24.198481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.198456    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-99668e-79c94cc98f-nndks"]
Oct 30 21:31:24.202478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.202461    4810 scope.go:122] "RemoveContainer" containerID="00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789"
Oct 30 21:31:24.202811 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:24.202792    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789\": container with ID starting with 00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789 not found: ID does not exist" containerID="00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789"
Oct 30 21:31:24.202936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.202821    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789"} err="failed to get container status \"00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789\": rpc error: code = NotFound desc = could not find container \"00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789\": container with ID starting with 00ec9e492e15a65d847d0fe27e8d71eb85b77f8d86dc489bd8de0f80a350c789 not found: ID does not exist"
Oct 30 21:31:24.202936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.202843    4810 scope.go:122] "RemoveContainer" containerID="1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407"
Oct 30 21:31:24.211576 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.211042    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"]
Oct 30 21:31:24.220957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.220928    4810 scope.go:122] "RemoveContainer" containerID="1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407"
Oct 30 21:31:24.223789 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:24.223741    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407\": container with ID starting with 1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407 not found: ID does not exist" containerID="1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407"
Oct 30 21:31:24.223950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.223798    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407"} err="failed to get container status \"1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407\": rpc error: code = NotFound desc = could not find container \"1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407\": container with ID starting with 1341aec4ba061cc4d657917a38481d214e873a8e596691db88d897c05bf3a407 not found: ID does not exist"
Oct 30 21:31:24.227985 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.227958    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-99668e-5d75c6c6c-9qtmt"]
Oct 30 21:31:24.467510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.467474    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"]
Oct 30 21:31:24.554936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.554898    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:31:24.567532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.567507    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:31:24.612838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.612803    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967070ba-c779-46ce-8256-cd93fab76c38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967070ba-c779-46ce-8256-cd93fab76c38-plugins-config-volume\") pod \"967070ba-c779-46ce-8256-cd93fab76c38\" (UID: \"967070ba-c779-46ce-8256-cd93fab76c38\") "
Oct 30 21:31:24.613277 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.613252    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967070ba-c779-46ce-8256-cd93fab76c38-plugins-config-volume" pod "967070ba-c779-46ce-8256-cd93fab76c38" (UID: "967070ba-c779-46ce-8256-cd93fab76c38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:24.613693 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.613668    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967070ba-c779-46ce-8256-cd93fab76c38-kube-api-access-cz9dm\" (UniqueName: \"kubernetes.io/projected/967070ba-c779-46ce-8256-cd93fab76c38-kube-api-access-cz9dm\") pod \"967070ba-c779-46ce-8256-cd93fab76c38\" (UID: \"967070ba-c779-46ce-8256-cd93fab76c38\") "
Oct 30 21:31:24.614235 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.614208    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967070ba-c779-46ce-8256-cd93fab76c38-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:24.618318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.617156    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967070ba-c779-46ce-8256-cd93fab76c38-kube-api-access-cz9dm" pod "967070ba-c779-46ce-8256-cd93fab76c38" (UID: "967070ba-c779-46ce-8256-cd93fab76c38"). InnerVolumeSpecName "kube-api-access-cz9dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:24.715374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.715291    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8c293f-f822-4033-99cc-0640e9013e20-kube-api-access-gf9dn\" (UniqueName: \"kubernetes.io/projected/7a8c293f-f822-4033-99cc-0640e9013e20-kube-api-access-gf9dn\") pod \"7a8c293f-f822-4033-99cc-0640e9013e20\" (UID: \"7a8c293f-f822-4033-99cc-0640e9013e20\") "
Oct 30 21:31:24.715374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.715366    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a8c293f-f822-4033-99cc-0640e9013e20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a8c293f-f822-4033-99cc-0640e9013e20-plugins-config-volume\") pod \"7a8c293f-f822-4033-99cc-0640e9013e20\" (UID: \"7a8c293f-f822-4033-99cc-0640e9013e20\") "
Oct 30 21:31:24.715791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.715757    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8c293f-f822-4033-99cc-0640e9013e20-plugins-config-volume" pod "7a8c293f-f822-4033-99cc-0640e9013e20" (UID: "7a8c293f-f822-4033-99cc-0640e9013e20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:24.716157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.716133    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz9dm\" (UniqueName: \"kubernetes.io/projected/967070ba-c779-46ce-8256-cd93fab76c38-kube-api-access-cz9dm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:24.716265 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.716161    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a8c293f-f822-4033-99cc-0640e9013e20-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:24.717580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.717559    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8c293f-f822-4033-99cc-0640e9013e20-kube-api-access-gf9dn" pod "7a8c293f-f822-4033-99cc-0640e9013e20" (UID: "7a8c293f-f822-4033-99cc-0640e9013e20"). InnerVolumeSpecName "kube-api-access-gf9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:24.821915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:24.818646    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf9dn\" (UniqueName: \"kubernetes.io/projected/7a8c293f-f822-4033-99cc-0640e9013e20-kube-api-access-gf9dn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:25.156821 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:25.156788    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:31:25.182917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.182885    4810 generic.go:356] "Generic (PLEG): container finished" podID="7a8c293f-f822-4033-99cc-0640e9013e20" containerID="2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840" exitCode=0
Oct 30 21:31:25.183171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.182914    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx" event={"ID":"7a8c293f-f822-4033-99cc-0640e9013e20","Type":"ContainerDied","Data":"2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840"}
Oct 30 21:31:25.183310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.182982    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"
Oct 30 21:31:25.183310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.183218    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx" event={"ID":"7a8c293f-f822-4033-99cc-0640e9013e20","Type":"ContainerDied","Data":"9dd6779869c0cf05e789a93c9a8eeec043ec0a0ffde5cdac0d65121df3c47422"}
Oct 30 21:31:25.183310 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.183262    4810 scope.go:122] "RemoveContainer" containerID="2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840"
Oct 30 21:31:25.184824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.184801    4810 generic.go:356] "Generic (PLEG): container finished" podID="967070ba-c779-46ce-8256-cd93fab76c38" containerID="8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2" exitCode=0
Oct 30 21:31:25.184957 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.184937    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6" event={"ID":"967070ba-c779-46ce-8256-cd93fab76c38","Type":"ContainerDied","Data":"8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2"}
Oct 30 21:31:25.185041 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.184979    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6" event={"ID":"967070ba-c779-46ce-8256-cd93fab76c38","Type":"ContainerDied","Data":"dc4515a47bb02f1f65d4f7b1b4e436de5f5e182c3097d088aa359240b722531e"}
Oct 30 21:31:25.185109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.185043    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"
Oct 30 21:31:25.200455 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.200433    4810 scope.go:122] "RemoveContainer" containerID="2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840"
Oct 30 21:31:25.200745 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:25.200723    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840\": container with ID starting with 2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840 not found: ID does not exist" containerID="2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840"
Oct 30 21:31:25.200803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.200753    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840"} err="failed to get container status \"2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840\": rpc error: code = NotFound desc = could not find container \"2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840\": container with ID starting with 2cf967447f00cdfdd653e286d04b96109cd7eb6af60ef7a7de47fbbe7185f840 not found: ID does not exist"
Oct 30 21:31:25.200803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.200770    4810 scope.go:122] "RemoveContainer" containerID="8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2"
Oct 30 21:31:25.212014 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.211988    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"]
Oct 30 21:31:25.214559 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.214528    4810 scope.go:122] "RemoveContainer" containerID="8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2"
Oct 30 21:31:25.215021 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:25.214996    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2\": container with ID starting with 8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2 not found: ID does not exist" containerID="8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2"
Oct 30 21:31:25.215101 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.215031    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2"} err="failed to get container status \"8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2\": rpc error: code = NotFound desc = could not find container \"8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2\": container with ID starting with 8a59920386b8d061d6aee67201d23484835e197331bd39f3537ff76c8ea1c4e2 not found: ID does not exist"
Oct 30 21:31:25.220270 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.220249    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e8063c-75f8dc5547-wqbsx"]
Oct 30 21:31:25.241655 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.241634    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"]
Oct 30 21:31:25.248018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.247997    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e8063c-749c5dc57-jf8b6"]
Oct 30 21:31:25.404894 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.404574    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"]
Oct 30 21:31:25.666285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.666246    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c" podUID="19991362-bc52-4564-b54e-64f087a20820" containerName="payload-processing" containerID="cri-o://b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6" gracePeriod=25
Oct 30 21:31:25.676600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.676569    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb1e381-15d1-4be5-ae3b-995385a56b10" path="/var/lib/kubelet/pods/3fb1e381-15d1-4be5-ae3b-995385a56b10/volumes"
Oct 30 21:31:25.677973 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.677714    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7" podUID="366d6f28-7835-4ea2-9ebf-32bfa47f3e0e" containerName="payload-pre-processing" containerID="cri-o://86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7" gracePeriod=25
Oct 30 21:31:25.680629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.680567    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8c293f-f822-4033-99cc-0640e9013e20" path="/var/lib/kubelet/pods/7a8c293f-f822-4033-99cc-0640e9013e20/volumes"
Oct 30 21:31:25.681486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.681433    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967070ba-c779-46ce-8256-cd93fab76c38" path="/var/lib/kubelet/pods/967070ba-c779-46ce-8256-cd93fab76c38/volumes"
Oct 30 21:31:25.682367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.682347    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a17faf1b-080e-4e34-8c63-e371f0dbe493" path="/var/lib/kubelet/pods/a17faf1b-080e-4e34-8c63-e371f0dbe493/volumes"
Oct 30 21:31:25.683721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:25.683701    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6089936-7fbd-42fa-b030-20e1af0a8e2b" path="/var/lib/kubelet/pods/e6089936-7fbd-42fa-b030-20e1af0a8e2b/volumes"
Oct 30 21:31:26.179624 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:26.179586    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"]
Oct 30 21:31:26.179833 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:26.179781    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy" containerID="cri-o://d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb" gracePeriod=30
Oct 30 21:31:26.219879 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:26.219304    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7" podUID="366d6f28-7835-4ea2-9ebf-32bfa47f3e0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:26.220213 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:26.220171    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c" podUID="19991362-bc52-4564-b54e-64f087a20820" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:27.106412 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:31:27.106383    4810 logging.go:55] [core] [Channel #12535 SubChannel #12536]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.117:50052", ServerName: "10.132.1.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.117:50052: connect: connection refused"
Oct 30 21:31:27.107134 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:31:27.107116    4810 logging.go:55] [core] [Channel #12537 SubChannel #12538]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.116:50052", ServerName: "10.132.1.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.116:50052: connect: connection refused"
Oct 30 21:31:27.224835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.222710    4810 generic.go:356] "Generic (PLEG): container finished" podID="a6c727b1-6fe8-46eb-95ff-4748528ee68d" containerID="128afce246826cbda2b3a0ec5b0a922a8f746654ede8d6869d97174b578e0049" exitCode=0
Oct 30 21:31:27.224835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.222911    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2" event={"ID":"a6c727b1-6fe8-46eb-95ff-4748528ee68d","Type":"ContainerDied","Data":"128afce246826cbda2b3a0ec5b0a922a8f746654ede8d6869d97174b578e0049"}
Oct 30 21:31:27.225652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.225306    4810 generic.go:356] "Generic (PLEG): container finished" podID="f268e880-2e55-4ff3-837d-a4fb2abbf8a1" containerID="d9d0439b665373fee670ec399653e8c1d39db1b98ad5a30a3a2548ece0f5f256" exitCode=0
Oct 30 21:31:27.225652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.225359    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj" event={"ID":"f268e880-2e55-4ff3-837d-a4fb2abbf8a1","Type":"ContainerDied","Data":"d9d0439b665373fee670ec399653e8c1d39db1b98ad5a30a3a2548ece0f5f256"}
Oct 30 21:31:27.343448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.343422    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:31:27.346955 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.346933    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:31:27.463673 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.463633    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-kube-api-access-47jrl\" (UniqueName: \"kubernetes.io/projected/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-kube-api-access-47jrl\") pod \"f268e880-2e55-4ff3-837d-a4fb2abbf8a1\" (UID: \"f268e880-2e55-4ff3-837d-a4fb2abbf8a1\") "
Oct 30 21:31:27.463900 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.463830    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-plugins-config-volume\") pod \"f268e880-2e55-4ff3-837d-a4fb2abbf8a1\" (UID: \"f268e880-2e55-4ff3-837d-a4fb2abbf8a1\") "
Oct 30 21:31:27.463985 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.463959    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c727b1-6fe8-46eb-95ff-4748528ee68d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c727b1-6fe8-46eb-95ff-4748528ee68d-plugins-config-volume\") pod \"a6c727b1-6fe8-46eb-95ff-4748528ee68d\" (UID: \"a6c727b1-6fe8-46eb-95ff-4748528ee68d\") "
Oct 30 21:31:27.464049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.464026    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c727b1-6fe8-46eb-95ff-4748528ee68d-kube-api-access-5wfd5\" (UniqueName: \"kubernetes.io/projected/a6c727b1-6fe8-46eb-95ff-4748528ee68d-kube-api-access-5wfd5\") pod \"a6c727b1-6fe8-46eb-95ff-4748528ee68d\" (UID: \"a6c727b1-6fe8-46eb-95ff-4748528ee68d\") "
Oct 30 21:31:27.464280 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.464182    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-plugins-config-volume" pod "f268e880-2e55-4ff3-837d-a4fb2abbf8a1" (UID: "f268e880-2e55-4ff3-837d-a4fb2abbf8a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:27.464401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.464373    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c727b1-6fe8-46eb-95ff-4748528ee68d-plugins-config-volume" pod "a6c727b1-6fe8-46eb-95ff-4748528ee68d" (UID: "a6c727b1-6fe8-46eb-95ff-4748528ee68d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:27.464647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.464627    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:27.464711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.464653    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c727b1-6fe8-46eb-95ff-4748528ee68d-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:27.466039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.466012    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-kube-api-access-47jrl" pod "f268e880-2e55-4ff3-837d-a4fb2abbf8a1" (UID: "f268e880-2e55-4ff3-837d-a4fb2abbf8a1"). InnerVolumeSpecName "kube-api-access-47jrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:27.466279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.466258    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c727b1-6fe8-46eb-95ff-4748528ee68d-kube-api-access-5wfd5" pod "a6c727b1-6fe8-46eb-95ff-4748528ee68d" (UID: "a6c727b1-6fe8-46eb-95ff-4748528ee68d"). InnerVolumeSpecName "kube-api-access-5wfd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:27.565415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.565347    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47jrl\" (UniqueName: \"kubernetes.io/projected/f268e880-2e55-4ff3-837d-a4fb2abbf8a1-kube-api-access-47jrl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:27.565415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.565374    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wfd5\" (UniqueName: \"kubernetes.io/projected/a6c727b1-6fe8-46eb-95ff-4748528ee68d-kube-api-access-5wfd5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:27.847298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.847213    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:31:27.946070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.946036    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"]
Oct 30 21:31:27.946269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:27.946245    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy" containerID="cri-o://deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473" gracePeriod=30
Oct 30 21:31:28.106126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.106031    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj" podUID="f268e880-2e55-4ff3-837d-a4fb2abbf8a1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.117:50052\" within 1s: context deadline exceeded"
Oct 30 21:31:28.106287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.106140    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:31:28.107074 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.107048    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2" podUID="a6c727b1-6fe8-46eb-95ff-4748528ee68d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.116:50052\" within 1s: context deadline exceeded"
Oct 30 21:31:28.107405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.107113    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:31:28.234651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.234612    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"
Oct 30 21:31:28.234651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.234629    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2" event={"ID":"a6c727b1-6fe8-46eb-95ff-4748528ee68d","Type":"ContainerDied","Data":"bd394428beaff7385d2c7c78aeac6a6c36ab48f00343ceb0d1fe6aa0d3363e67"}
Oct 30 21:31:28.234940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.234697    4810 scope.go:122] "RemoveContainer" containerID="128afce246826cbda2b3a0ec5b0a922a8f746654ede8d6869d97174b578e0049"
Oct 30 21:31:28.236645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.236628    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"
Oct 30 21:31:28.236959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.236625    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj" event={"ID":"f268e880-2e55-4ff3-837d-a4fb2abbf8a1","Type":"ContainerDied","Data":"ba050f29b95b10f82d1d8ea53aa4ee1fffaba1860b8123624a12ee9c41e94436"}
Oct 30 21:31:28.246494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.246471    4810 scope.go:122] "RemoveContainer" containerID="d9d0439b665373fee670ec399653e8c1d39db1b98ad5a30a3a2548ece0f5f256"
Oct 30 21:31:28.263230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.263207    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"]
Oct 30 21:31:28.266532 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.266510    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5506f4-7587579679-r6sfj"]
Oct 30 21:31:28.277099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.277076    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"]
Oct 30 21:31:28.279782 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.279756    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5506f4-6db66ccddc-gg2k2"]
Oct 30 21:31:28.379679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.379650    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"]
Oct 30 21:31:28.380267 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380254    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d9b133d7-be5b-4088-80aa-648b7c4c59ad" containerName="revoke-keys"
Oct 30 21:31:28.380328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380269    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b133d7-be5b-4088-80aa-648b7c4c59ad" containerName="revoke-keys"
Oct 30 21:31:28.380328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380280    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3fb1e381-15d1-4be5-ae3b-995385a56b10" containerName="payload-pre-processing"
Oct 30 21:31:28.380328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380285    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb1e381-15d1-4be5-ae3b-995385a56b10" containerName="payload-pre-processing"
Oct 30 21:31:28.380328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380296    4810 cpu_manager.go:414] "containerMap: removing container" podUID="967070ba-c779-46ce-8256-cd93fab76c38" containerName="payload-processing"
Oct 30 21:31:28.380328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380301    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967070ba-c779-46ce-8256-cd93fab76c38" containerName="payload-processing"
Oct 30 21:31:28.380328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380322    4810 cpu_manager.go:414] "containerMap: removing container" podUID="19a5d622-b35e-4899-a5d4-a37a0bcff6dd" containerName="revoke-keys"
Oct 30 21:31:28.380328 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380327    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a5d622-b35e-4899-a5d4-a37a0bcff6dd" containerName="revoke-keys"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380334    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7a8c293f-f822-4033-99cc-0640e9013e20" containerName="payload-pre-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380340    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8c293f-f822-4033-99cc-0640e9013e20" containerName="payload-pre-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380353    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e6089936-7fbd-42fa-b030-20e1af0a8e2b" containerName="payload-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380358    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6089936-7fbd-42fa-b030-20e1af0a8e2b" containerName="payload-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380375    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f268e880-2e55-4ff3-837d-a4fb2abbf8a1" containerName="payload-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380380    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f268e880-2e55-4ff3-837d-a4fb2abbf8a1" containerName="payload-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380395    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a6c727b1-6fe8-46eb-95ff-4748528ee68d" containerName="payload-pre-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380400    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c727b1-6fe8-46eb-95ff-4748528ee68d" containerName="payload-pre-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380479    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="967070ba-c779-46ce-8256-cd93fab76c38" containerName="payload-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380489    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a5d622-b35e-4899-a5d4-a37a0bcff6dd" containerName="revoke-keys"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380495    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8c293f-f822-4033-99cc-0640e9013e20" containerName="payload-pre-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380502    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6089936-7fbd-42fa-b030-20e1af0a8e2b" containerName="payload-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380511    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f268e880-2e55-4ff3-837d-a4fb2abbf8a1" containerName="payload-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380518    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c727b1-6fe8-46eb-95ff-4748528ee68d" containerName="payload-pre-processing"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380526    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b133d7-be5b-4088-80aa-648b7c4c59ad" containerName="revoke-keys"
Oct 30 21:31:28.380548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.380532    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb1e381-15d1-4be5-ae3b-995385a56b10" containerName="payload-pre-processing"
Oct 30 21:31:28.385799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.385781    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"]
Oct 30 21:31:28.385966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.385946    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.389868 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.389834    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.393063 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.393040    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f8e2c3-openshift-default-dockercfg-b259x\""
Oct 30 21:31:28.395592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.395300    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"]
Oct 30 21:31:28.403444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.403422    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"]
Oct 30 21:31:28.474380 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474350    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-token\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.474531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474389    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-data\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.474531 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474439    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-certs\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.474625 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474562    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b01c180f-27pzz\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.474625 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474600    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b01c180f-27pzz\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.474758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474658    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-credential-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.474758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474744    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-podinfo\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.474824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474782    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istiod-ca-cert\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.474824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474809    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjdh8\" (UniqueName: \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-kube-api-access-pjdh8\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.474950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474929    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6nnl\" (UniqueName: \"kubernetes.io/projected/3d6c4c8b-8e90-4f30-a445-35ee31746760-kube-api-access-l6nnl\") pod \"maas-api-revoke-sub-b01c180f-27pzz\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.474998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.474986    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.475060 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.475047    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-envoy\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.493417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.493398    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:28.493569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.493553    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:31:28.576714 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.576675    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6nnl\" (UniqueName: \"kubernetes.io/projected/3d6c4c8b-8e90-4f30-a445-35ee31746760-kube-api-access-l6nnl\") pod \"maas-api-revoke-sub-b01c180f-27pzz\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.577058 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.577033    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.577206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.577190    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-envoy\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.577337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.577324    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-token\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.580397    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-data\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.580473    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-certs\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.580514    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b01c180f-27pzz\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.580575    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b01c180f-27pzz\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.580694    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-credential-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.580781    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-podinfo\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.580848    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istiod-ca-cert\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.580929    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjdh8\" (UniqueName: \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-kube-api-access-pjdh8\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.581622    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-data\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.581984    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-certs\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.585738    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b01c180f-27pzz\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.586700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.586377    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b01c180f-27pzz\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.587471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.586911    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-credential-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.605878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.588185    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.605878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.591640    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istiod-ca-cert\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.605878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.598388    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-envoy\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.605878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.598845    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-podinfo\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.615893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.612434    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6nnl\" (UniqueName: \"kubernetes.io/projected/3d6c4c8b-8e90-4f30-a445-35ee31746760-kube-api-access-l6nnl\") pod \"maas-api-revoke-sub-b01c180f-27pzz\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.616502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.616476    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjdh8\" (UniqueName: \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-kube-api-access-pjdh8\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.618088 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.618060    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-token\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.716325 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.716200    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:28.720434 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.720396    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:28.873842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.871418    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"]
Oct 30 21:31:28.877183 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:28.877137    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-00909f-pskdh"]
Oct 30 21:31:29.068113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.068039    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"]
Oct 30 21:31:29.086995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.086961    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:29.087275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.087196    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:29.087465 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.087423    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:29.112470 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.112423    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"]
Oct 30 21:31:29.247300 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.247266    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz" event={"ID":"3d6c4c8b-8e90-4f30-a445-35ee31746760","Type":"ContainerStarted","Data":"1a87ffc4ec173e3411a5ba09b3ddb26b811d7fb72c947295abd4482a734e5969"}
Oct 30 21:31:29.250316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.250283    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" event={"ID":"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7","Type":"ContainerStarted","Data":"df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0"}
Oct 30 21:31:29.250472 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.250337    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" event={"ID":"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7","Type":"ContainerStarted","Data":"7b50326c30bd633bce9361412d67ba591b44ba4bd8d5dc25617838e0b4c71bbf"}
Oct 30 21:31:29.273779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.273706    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" podStartSLOduration=1.273688219 podStartE2EDuration="1.273688219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:29.270542756 +0000 UTC m=+3255.762359303" watchObservedRunningTime="2026-10-30 21:31:29.273688219 +0000 UTC m=+3255.765504766"
Oct 30 21:31:29.279813 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.279711    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"]
Oct 30 21:31:29.284566 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.284537    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e07442-wzm8n"]
Oct 30 21:31:29.468632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.468573    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8" podUID="b258a27c-5663-494e-abff-b1979edfee61" containerName="payload-processing" containerID="cri-o://bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796" gracePeriod=25
Oct 30 21:31:29.667902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.667794    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a5d622-b35e-4899-a5d4-a37a0bcff6dd" path="/var/lib/kubelet/pods/19a5d622-b35e-4899-a5d4-a37a0bcff6dd/volumes"
Oct 30 21:31:29.668508 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.668485    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c727b1-6fe8-46eb-95ff-4748528ee68d" path="/var/lib/kubelet/pods/a6c727b1-6fe8-46eb-95ff-4748528ee68d/volumes"
Oct 30 21:31:29.669175 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.669152    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b133d7-be5b-4088-80aa-648b7c4c59ad" path="/var/lib/kubelet/pods/d9b133d7-be5b-4088-80aa-648b7c4c59ad/volumes"
Oct 30 21:31:29.669985 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.669963    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f268e880-2e55-4ff3-837d-a4fb2abbf8a1" path="/var/lib/kubelet/pods/f268e880-2e55-4ff3-837d-a4fb2abbf8a1/volumes"
Oct 30 21:31:29.717262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:29.717222    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:30.029530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:30.029437    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"]
Oct 30 21:31:30.033442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:30.033418    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa325ed3-7w5gz"]
Oct 30 21:31:30.260713 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:30.260673    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz" event={"ID":"3d6c4c8b-8e90-4f30-a445-35ee31746760","Type":"ContainerStarted","Data":"f4b3bebb1738463fe2d1d78d24cf57d165e1b25d026055918984d03f87ea106a"}
Oct 30 21:31:30.318903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:30.318815    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz" podStartSLOduration=2.318792127 podStartE2EDuration="2.318792127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:30.278390469 +0000 UTC m=+3256.770207015" watchObservedRunningTime="2026-10-30 21:31:30.318792127 +0000 UTC m=+3256.810608673"
Oct 30 21:31:30.405132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:30.405073    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6" podUID="ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1" containerName="payload-pre-processing" containerID="cri-o://542c215b15de66e055e780fbc4d641ac62c45950d66c7ac92871400bd0221fa7" gracePeriod=25
Oct 30 21:31:30.718254 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:30.718221    4810 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Oct 30 21:31:30.718425 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:30.718291    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Oct 30 21:31:31.024627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.024528    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"]
Oct 30 21:31:31.027923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.027845    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f513ec-sjt6h"]
Oct 30 21:31:31.257666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.257630    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6" podUID="ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:31.304935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.286366    4810 generic.go:356] "Generic (PLEG): container finished" podID="3d6c4c8b-8e90-4f30-a445-35ee31746760" containerID="f4b3bebb1738463fe2d1d78d24cf57d165e1b25d026055918984d03f87ea106a" exitCode=0
Oct 30 21:31:31.304935 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.287758    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz" event={"ID":"3d6c4c8b-8e90-4f30-a445-35ee31746760","Type":"ContainerDied","Data":"f4b3bebb1738463fe2d1d78d24cf57d165e1b25d026055918984d03f87ea106a"}
Oct 30 21:31:31.537043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.537017    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:31:31.637133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.637101    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-istio-token\" (UniqueName: \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-istio-token\") pod \"39578b95-1622-484d-b298-366283d7c78d\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") "
Oct 30 21:31:31.637309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.637202    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39578b95-1622-484d-b298-366283d7c78d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39578b95-1622-484d-b298-366283d7c78d-istio-podinfo\") pod \"39578b95-1622-484d-b298-366283d7c78d\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") "
Oct 30 21:31:31.637309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.637225    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-socket\") pod \"39578b95-1622-484d-b298-366283d7c78d\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") "
Oct 30 21:31:31.637426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.637386    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-envoy\") pod \"39578b95-1622-484d-b298-366283d7c78d\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") "
Oct 30 21:31:31.637426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.637412    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-kube-api-access-lwbqj\" (UniqueName: \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-kube-api-access-lwbqj\") pod \"39578b95-1622-484d-b298-366283d7c78d\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") "
Oct 30 21:31:31.637529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.637433    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39578b95-1622-484d-b298-366283d7c78d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39578b95-1622-484d-b298-366283d7c78d-istiod-ca-cert\") pod \"39578b95-1622-484d-b298-366283d7c78d\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") "
Oct 30 21:31:31.637529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.637470    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-certs\") pod \"39578b95-1622-484d-b298-366283d7c78d\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") "
Oct 30 21:31:31.637529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.637494    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-credential-socket\") pod \"39578b95-1622-484d-b298-366283d7c78d\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") "
Oct 30 21:31:31.637529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.637527    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-data\") pod \"39578b95-1622-484d-b298-366283d7c78d\" (UID: \"39578b95-1622-484d-b298-366283d7c78d\") "
Oct 30 21:31:31.638717 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.638500    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-certs" pod "39578b95-1622-484d-b298-366283d7c78d" (UID: "39578b95-1622-484d-b298-366283d7c78d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:31.638852 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.638728    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-socket" pod "39578b95-1622-484d-b298-366283d7c78d" (UID: "39578b95-1622-484d-b298-366283d7c78d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:31.638948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.638840    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39578b95-1622-484d-b298-366283d7c78d-istiod-ca-cert" pod "39578b95-1622-484d-b298-366283d7c78d" (UID: "39578b95-1622-484d-b298-366283d7c78d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:31.639002 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.638937    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-data" pod "39578b95-1622-484d-b298-366283d7c78d" (UID: "39578b95-1622-484d-b298-366283d7c78d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:31.640054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.640011    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-credential-socket" pod "39578b95-1622-484d-b298-366283d7c78d" (UID: "39578b95-1622-484d-b298-366283d7c78d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:31.641671 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.641558    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-istio-token" pod "39578b95-1622-484d-b298-366283d7c78d" (UID: "39578b95-1622-484d-b298-366283d7c78d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:31.642831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.642792    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-kube-api-access-lwbqj" pod "39578b95-1622-484d-b298-366283d7c78d" (UID: "39578b95-1622-484d-b298-366283d7c78d"). InnerVolumeSpecName "kube-api-access-lwbqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:31.644327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.644302    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-envoy" pod "39578b95-1622-484d-b298-366283d7c78d" (UID: "39578b95-1622-484d-b298-366283d7c78d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:31.646150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.646112    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39578b95-1622-484d-b298-366283d7c78d-istio-podinfo" pod "39578b95-1622-484d-b298-366283d7c78d" (UID: "39578b95-1622-484d-b298-366283d7c78d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:31:31.663444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.663422    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56dce06c-45c2-476f-aa9d-ca0bfe42c86d" path="/var/lib/kubelet/pods/56dce06c-45c2-476f-aa9d-ca0bfe42c86d/volumes"
Oct 30 21:31:31.663909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.663894    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f769b235-7234-4aee-9a42-3a49abcb6594" path="/var/lib/kubelet/pods/f769b235-7234-4aee-9a42-3a49abcb6594/volumes"
Oct 30 21:31:31.717218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.717187    4810 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Oct 30 21:31:31.717375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.717240    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Oct 30 21:31:31.740317 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.740292    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:31.740317 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.740318    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwbqj\" (UniqueName: \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-kube-api-access-lwbqj\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:31.740502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.740333    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39578b95-1622-484d-b298-366283d7c78d-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:31.740502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.740347    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:31.740502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.740361    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:31.740502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.740373    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:31.740502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.740387    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39578b95-1622-484d-b298-366283d7c78d-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:31.740502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.740400    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39578b95-1622-484d-b298-366283d7c78d-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:31.740502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:31.740412    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39578b95-1622-484d-b298-366283d7c78d-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:32.306007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.305974    4810 generic.go:356] "Generic (PLEG): container finished" podID="39578b95-1622-484d-b298-366283d7c78d" containerID="d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb" exitCode=0
Oct 30 21:31:32.306501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.306045    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"
Oct 30 21:31:32.306501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.306192    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" event={"ID":"39578b95-1622-484d-b298-366283d7c78d","Type":"ContainerDied","Data":"d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb"}
Oct 30 21:31:32.306501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.306243    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch" event={"ID":"39578b95-1622-484d-b298-366283d7c78d","Type":"ContainerDied","Data":"21df41ef463393d6014b9e3b0290e085bd5c29fb86510866aeb7156d95c7098b"}
Oct 30 21:31:32.306501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.306272    4810 scope.go:122] "RemoveContainer" containerID="d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb"
Oct 30 21:31:32.320065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.320024    4810 scope.go:122] "RemoveContainer" containerID="d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb"
Oct 30 21:31:32.320391 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:32.320337    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb\": container with ID starting with d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb not found: ID does not exist" containerID="d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb"
Oct 30 21:31:32.320391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.320368    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb"} err="failed to get container status \"d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb\": rpc error: code = NotFound desc = could not find container \"d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb\": container with ID starting with d9dceccda6972ffa7ebe916bbcdd269055128bfd31e9fead93789f35994975eb not found: ID does not exist"
Oct 30 21:31:32.333408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.332439    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"]
Oct 30 21:31:32.336174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.336152    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-9cqch"]
Oct 30 21:31:32.660181 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.660156    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:32.728646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.722265    4810 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Oct 30 21:31:32.728646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.722321    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Oct 30 21:31:32.761515 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.761482    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6c4c8b-8e90-4f30-a445-35ee31746760-kube-api-access-l6nnl\" (UniqueName: \"kubernetes.io/projected/3d6c4c8b-8e90-4f30-a445-35ee31746760-kube-api-access-l6nnl\") pod \"3d6c4c8b-8e90-4f30-a445-35ee31746760\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") "
Oct 30 21:31:32.761686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.761618    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-xks-ca\") pod \"3d6c4c8b-8e90-4f30-a445-35ee31746760\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") "
Oct 30 21:31:32.761779 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.761710    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-openshift-ca\") pod \"3d6c4c8b-8e90-4f30-a445-35ee31746760\" (UID: \"3d6c4c8b-8e90-4f30-a445-35ee31746760\") "
Oct 30 21:31:32.762113 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.762087    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-openshift-ca" pod "3d6c4c8b-8e90-4f30-a445-35ee31746760" (UID: "3d6c4c8b-8e90-4f30-a445-35ee31746760"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:32.762398 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.762376    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:32.763931 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.763903    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-xks-ca" pod "3d6c4c8b-8e90-4f30-a445-35ee31746760" (UID: "3d6c4c8b-8e90-4f30-a445-35ee31746760"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:32.765028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.764989    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6c4c8b-8e90-4f30-a445-35ee31746760-kube-api-access-l6nnl" pod "3d6c4c8b-8e90-4f30-a445-35ee31746760" (UID: "3d6c4c8b-8e90-4f30-a445-35ee31746760"). InnerVolumeSpecName "kube-api-access-l6nnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:32.863778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.863737    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6c4c8b-8e90-4f30-a445-35ee31746760-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:32.863778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.863779    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6nnl\" (UniqueName: \"kubernetes.io/projected/3d6c4c8b-8e90-4f30-a445-35ee31746760-kube-api-access-l6nnl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:32.942677 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.942586    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"]
Oct 30 21:31:32.943281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.943261    4810 cpu_manager.go:414] "containerMap: removing container" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy"
Oct 30 21:31:32.943281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.943280    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy"
Oct 30 21:31:32.943442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.943318    4810 cpu_manager.go:414] "containerMap: removing container" podUID="3d6c4c8b-8e90-4f30-a445-35ee31746760" containerName="revoke-keys"
Oct 30 21:31:32.943442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.943323    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6c4c8b-8e90-4f30-a445-35ee31746760" containerName="revoke-keys"
Oct 30 21:31:32.943442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.943435    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6c4c8b-8e90-4f30-a445-35ee31746760" containerName="revoke-keys"
Oct 30 21:31:32.943581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.943450    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="39578b95-1622-484d-b298-366283d7c78d" containerName="istio-proxy"
Oct 30 21:31:32.947420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.947378    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:32.951768 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:32.951581    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"]
Oct 30 21:31:33.066468 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.066443    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4qjw\" (UniqueName: \"kubernetes.io/projected/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-kube-api-access-j4qjw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:33.066593 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.066514    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:33.066713 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.066686    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:33.169350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.168309    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:33.169350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.168428    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4qjw\" (UniqueName: \"kubernetes.io/projected/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-kube-api-access-j4qjw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:33.169350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.168476    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:33.169350 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.169297    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:33.183883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.178810    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:33.204373 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.204273    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4qjw\" (UniqueName: \"kubernetes.io/projected/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-kube-api-access-j4qjw\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:33.258034 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.258008    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 21:31:33.265042 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.265021    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:33.339246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.339212    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz" event={"ID":"3d6c4c8b-8e90-4f30-a445-35ee31746760","Type":"ContainerDied","Data":"1a87ffc4ec173e3411a5ba09b3ddb26b811d7fb72c947295abd4482a734e5969"}
Oct 30 21:31:33.339626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.339260    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a87ffc4ec173e3411a5ba09b3ddb26b811d7fb72c947295abd4482a734e5969"
Oct 30 21:31:33.339626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.339320    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b01c180f-27pzz"
Oct 30 21:31:33.341405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.341374    4810 generic.go:356] "Generic (PLEG): container finished" podID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerID="deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473" exitCode=0
Oct 30 21:31:33.341591 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.341574    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" event={"ID":"5a6887c4-2c11-40f4-87f4-1db9dc258170","Type":"ContainerDied","Data":"deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473"}
Oct 30 21:31:33.341659 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.341621    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd" event={"ID":"5a6887c4-2c11-40f4-87f4-1db9dc258170","Type":"ContainerDied","Data":"8c699f7add2b3fabc338b03947f3fad5a68b32f18c445dd878dba33128e92ccd"}
Oct 30 21:31:33.341732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.341668    4810 scope.go:122] "RemoveContainer" containerID="deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473"
Oct 30 21:31:33.341812 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.341798    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"
Oct 30 21:31:33.364246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.364225    4810 scope.go:122] "RemoveContainer" containerID="deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473"
Oct 30 21:31:33.364727 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:33.364699    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473\": container with ID starting with deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473 not found: ID does not exist" containerID="deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473"
Oct 30 21:31:33.364823 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.364740    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473"} err="failed to get container status \"deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473\": rpc error: code = NotFound desc = could not find container \"deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473\": container with ID starting with deb40544878344802a9a3a9eafea3de0eed705c98a5834777e6319f661a7d473 not found: ID does not exist"
Oct 30 21:31:33.375478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.375448    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-data\") pod \"5a6887c4-2c11-40f4-87f4-1db9dc258170\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") "
Oct 30 21:31:33.381431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.375913    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-data" pod "5a6887c4-2c11-40f4-87f4-1db9dc258170" (UID: "5a6887c4-2c11-40f4-87f4-1db9dc258170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:33.381724 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.381706    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-token\" (UniqueName: \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-token\") pod \"5a6887c4-2c11-40f4-87f4-1db9dc258170\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") "
Oct 30 21:31:33.382056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.382036    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-socket\") pod \"5a6887c4-2c11-40f4-87f4-1db9dc258170\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") "
Oct 30 21:31:33.402336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.402181    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-socket" pod "5a6887c4-2c11-40f4-87f4-1db9dc258170" (UID: "5a6887c4-2c11-40f4-87f4-1db9dc258170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:33.402963 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.402915    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6887c4-2c11-40f4-87f4-1db9dc258170-istiod-ca-cert" pod "5a6887c4-2c11-40f4-87f4-1db9dc258170" (UID: "5a6887c4-2c11-40f4-87f4-1db9dc258170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:33.403111 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.402272    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6887c4-2c11-40f4-87f4-1db9dc258170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6887c4-2c11-40f4-87f4-1db9dc258170-istiod-ca-cert\") pod \"5a6887c4-2c11-40f4-87f4-1db9dc258170\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") "
Oct 30 21:31:33.403581 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.403536    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-certs" pod "5a6887c4-2c11-40f4-87f4-1db9dc258170" (UID: "5a6887c4-2c11-40f4-87f4-1db9dc258170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:33.403724 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.403707    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-certs\") pod \"5a6887c4-2c11-40f4-87f4-1db9dc258170\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") "
Oct 30 21:31:33.404441 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.404422    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-podinfo\") pod \"5a6887c4-2c11-40f4-87f4-1db9dc258170\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") "
Oct 30 21:31:33.404902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.404849    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-credential-socket" pod "5a6887c4-2c11-40f4-87f4-1db9dc258170" (UID: "5a6887c4-2c11-40f4-87f4-1db9dc258170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:33.405069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.405050    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-credential-socket\") pod \"5a6887c4-2c11-40f4-87f4-1db9dc258170\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") "
Oct 30 21:31:33.405218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.405204    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-kube-api-access-bpkfk\" (UniqueName: \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-kube-api-access-bpkfk\") pod \"5a6887c4-2c11-40f4-87f4-1db9dc258170\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") "
Oct 30 21:31:33.408383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.408364    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-envoy\") pod \"5a6887c4-2c11-40f4-87f4-1db9dc258170\" (UID: \"5a6887c4-2c11-40f4-87f4-1db9dc258170\") "
Oct 30 21:31:33.409388 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.409367    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:33.409539 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.409523    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6887c4-2c11-40f4-87f4-1db9dc258170-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:33.416770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.415373    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:33.416770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.415394    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:33.416770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.415407    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:33.426606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.426578    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-envoy" pod "5a6887c4-2c11-40f4-87f4-1db9dc258170" (UID: "5a6887c4-2c11-40f4-87f4-1db9dc258170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:33.426738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.426721    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-kube-api-access-bpkfk" pod "5a6887c4-2c11-40f4-87f4-1db9dc258170" (UID: "5a6887c4-2c11-40f4-87f4-1db9dc258170"). InnerVolumeSpecName "kube-api-access-bpkfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:33.426837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.426820    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-token" pod "5a6887c4-2c11-40f4-87f4-1db9dc258170" (UID: "5a6887c4-2c11-40f4-87f4-1db9dc258170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:33.435838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.435790    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-podinfo" pod "5a6887c4-2c11-40f4-87f4-1db9dc258170" (UID: "5a6887c4-2c11-40f4-87f4-1db9dc258170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:31:33.516990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.516791    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:33.516990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.516834    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpkfk\" (UniqueName: \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-kube-api-access-bpkfk\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:33.516990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.516871    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:33.516990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.516888    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6887c4-2c11-40f4-87f4-1db9dc258170-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:33.635065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.634732    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"]
Oct 30 21:31:33.667318 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.667290    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39578b95-1622-484d-b298-366283d7c78d" path="/var/lib/kubelet/pods/39578b95-1622-484d-b298-366283d7c78d/volumes"
Oct 30 21:31:33.723875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.723173    4810 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Oct 30 21:31:33.723875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.723230    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Oct 30 21:31:33.801062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.800930    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"]
Oct 30 21:31:33.836880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:33.818195    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-b7dfd"]
Oct 30 21:31:34.357887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:34.357777    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn" event={"ID":"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5","Type":"ContainerStarted","Data":"6922e4787f4b104de85b2b9f773fa907c335143388c672275e9875cf679f02be"}
Oct 30 21:31:34.357887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:34.357831    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn" event={"ID":"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5","Type":"ContainerStarted","Data":"1758330a36da9a36cfb13a1e188eed2b4d012d3863aaedfb2b23757e6d60ac53"}
Oct 30 21:31:34.383886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:34.377476    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn" podStartSLOduration=2.377457557 podStartE2EDuration="2.377457557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:34.376056874 +0000 UTC m=+3260.867873420" watchObservedRunningTime="2026-10-30 21:31:34.377457557 +0000 UTC m=+3260.869274101"
Oct 30 21:31:34.722157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:34.722073    4810 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Oct 30 21:31:34.722443 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:34.722399    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Oct 30 21:31:35.368341 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:35.368298    4810 generic.go:356] "Generic (PLEG): container finished" podID="24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5" containerID="6922e4787f4b104de85b2b9f773fa907c335143388c672275e9875cf679f02be" exitCode=0
Oct 30 21:31:35.368791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:35.368379    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn" event={"ID":"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5","Type":"ContainerDied","Data":"6922e4787f4b104de85b2b9f773fa907c335143388c672275e9875cf679f02be"}
Oct 30 21:31:35.664745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:35.664711    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" path="/var/lib/kubelet/pods/5a6887c4-2c11-40f4-87f4-1db9dc258170/volumes"
Oct 30 21:31:35.722429 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:35.722396    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:35.722885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:35.722867    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:35.723545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:35.723525    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:31:36.217806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.217767    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7" podUID="366d6f28-7835-4ea2-9ebf-32bfa47f3e0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:36.217990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.217767    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c" podUID="19991362-bc52-4564-b54e-64f087a20820" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:36.559578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.559546    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:36.672584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.672549    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-kube-api-access-j4qjw\" (UniqueName: \"kubernetes.io/projected/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-kube-api-access-j4qjw\") pod \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") "
Oct 30 21:31:36.672765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.672711    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-xks-ca\") pod \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") "
Oct 30 21:31:36.672765 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.672743    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-openshift-ca\") pod \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\" (UID: \"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5\") "
Oct 30 21:31:36.673172 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.673147    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-openshift-ca" pod "24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5" (UID: "24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:36.673289 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.673267    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:36.674817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.674795    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-xks-ca" pod "24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5" (UID: "24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:36.674921 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.674902    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-kube-api-access-j4qjw" pod "24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5" (UID: "24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5"). InnerVolumeSpecName "kube-api-access-j4qjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:36.774450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.774395    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:36.774450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:36.774436    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4qjw\" (UniqueName: \"kubernetes.io/projected/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5-kube-api-access-j4qjw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:37.389942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:37.389912    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"
Oct 30 21:31:37.390146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:37.389912    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn" event={"ID":"24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5","Type":"ContainerDied","Data":"1758330a36da9a36cfb13a1e188eed2b4d012d3863aaedfb2b23757e6d60ac53"}
Oct 30 21:31:37.390146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:37.390053    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1758330a36da9a36cfb13a1e188eed2b4d012d3863aaedfb2b23757e6d60ac53"
Oct 30 21:31:38.310198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.310154    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"]
Oct 30 21:31:38.311128 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.311108    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy"
Oct 30 21:31:38.311185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.311131    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy"
Oct 30 21:31:38.311218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.311189    4810 cpu_manager.go:414] "containerMap: removing container" podUID="24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5" containerName="revoke-keys"
Oct 30 21:31:38.311218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.311198    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5" containerName="revoke-keys"
Oct 30 21:31:38.311384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.311342    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5" containerName="revoke-keys"
Oct 30 21:31:38.311384 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.311364    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6887c4-2c11-40f4-87f4-1db9dc258170" containerName="istio-proxy"
Oct 30 21:31:38.315746 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.315725    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.322697 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.322670    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:31:38.324891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.324833    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:38.327497 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.327473    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"]
Oct 30 21:31:38.394265 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.394231    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74cn4\" (UniqueName: \"kubernetes.io/projected/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-kube-api-access-74cn4\") pod \"maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.394471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.394305    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.394471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.394394    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.495909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.495812    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74cn4\" (UniqueName: \"kubernetes.io/projected/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-kube-api-access-74cn4\") pod \"maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.496130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.495965    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.496130 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.495987    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.496936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.496910    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.498402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.498378    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.505008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.504982    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74cn4\" (UniqueName: \"kubernetes.io/projected/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-kube-api-access-74cn4\") pod \"maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.527997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.527969    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"]
Oct 30 21:31:38.532301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.532285    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.542206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.542175    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"]
Oct 30 21:31:38.597336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.597244    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.597513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.597369    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.597513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.597438    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfp9d\" (UniqueName: \"kubernetes.io/projected/b7b0e127-0fa4-436f-9599-8dc8081a3c62-kube-api-access-cfp9d\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.633471 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.633436    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:38.698643 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.698613    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.698747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.698730    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.698803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.698784    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfp9d\" (UniqueName: \"kubernetes.io/projected/b7b0e127-0fa4-436f-9599-8dc8081a3c62-kube-api-access-cfp9d\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.700222 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.700193    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.712116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.703978    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.723337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.723304    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfp9d\" (UniqueName: \"kubernetes.io/projected/b7b0e127-0fa4-436f-9599-8dc8081a3c62-kube-api-access-cfp9d\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.848648 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.848561    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:38.924915 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:31:38.924641    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2835f0_fa2c_4a09_b169_5ca8c6a31b14.slice/crio-aee71f7555239d48246d80b3e7f923da4832a15cf2c246580ca127fafc22d18e WatchSource:0}: Error finding container aee71f7555239d48246d80b3e7f923da4832a15cf2c246580ca127fafc22d18e: Status 404 returned error can't find the container with id aee71f7555239d48246d80b3e7f923da4832a15cf2c246580ca127fafc22d18e
Oct 30 21:31:38.926578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:38.926554    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"]
Oct 30 21:31:39.127185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.127111    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"]
Oct 30 21:31:39.184882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.180944    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"]
Oct 30 21:31:39.275145 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.275098    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8" podUID="b258a27c-5663-494e-abff-b1979edfee61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:39.413119 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.413072    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf" event={"ID":"b7b0e127-0fa4-436f-9599-8dc8081a3c62","Type":"ContainerStarted","Data":"a44f423e73b66dcd040048d7ae307d9a0ff415ae0fe0cb332e93d6e6ef193505"}
Oct 30 21:31:39.415637 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.415607    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb" event={"ID":"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14","Type":"ContainerStarted","Data":"79af9bccb54f92d55ce6d8e0bd0799846701f790ea2543e1d9c40c5628f2397d"}
Oct 30 21:31:39.415766 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.415658    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb" event={"ID":"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14","Type":"ContainerStarted","Data":"aee71f7555239d48246d80b3e7f923da4832a15cf2c246580ca127fafc22d18e"}
Oct 30 21:31:39.455342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.455282    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb" podStartSLOduration=1.455240606 podStartE2EDuration="1.455240606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:39.4527475 +0000 UTC m=+3265.944564056" watchObservedRunningTime="2026-10-30 21:31:39.455240606 +0000 UTC m=+3265.947057149"
Oct 30 21:31:39.583218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.583184    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"]
Oct 30 21:31:39.594807 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.594110    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:39.596198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.596173    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"]
Oct 30 21:31:39.623562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.623526    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b07a74d0-q2c4g\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:39.623978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.623958    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b07a74d0-q2c4g\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:39.624249 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.624214    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9npwb\" (UniqueName: \"kubernetes.io/projected/54769587-dc35-44e4-9f79-7701a79f4aa0-kube-api-access-9npwb\") pod \"maas-api-revoke-sub-b07a74d0-q2c4g\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:39.726083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.725998    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b07a74d0-q2c4g\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:39.726083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.726038    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b07a74d0-q2c4g\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:39.726083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.726061    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9npwb\" (UniqueName: \"kubernetes.io/projected/54769587-dc35-44e4-9f79-7701a79f4aa0-kube-api-access-9npwb\") pod \"maas-api-revoke-sub-b07a74d0-q2c4g\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:39.726651 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.726626    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b07a74d0-q2c4g\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:39.728504 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.728480    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b07a74d0-q2c4g\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:39.736751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.736724    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9npwb\" (UniqueName: \"kubernetes.io/projected/54769587-dc35-44e4-9f79-7701a79f4aa0-kube-api-access-9npwb\") pod \"maas-api-revoke-sub-b07a74d0-q2c4g\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:39.913965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:39.913924    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:40.208583 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:40.208553    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"]
Oct 30 21:31:40.210386 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:31:40.210360    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54769587_dc35_44e4_9f79_7701a79f4aa0.slice/crio-ca1886235b9626c510ac70e091d6b5b9007f9435f9447dc87c87ab4ca6fce8ba WatchSource:0}: Error finding container ca1886235b9626c510ac70e091d6b5b9007f9435f9447dc87c87ab4ca6fce8ba: Status 404 returned error can't find the container with id ca1886235b9626c510ac70e091d6b5b9007f9435f9447dc87c87ab4ca6fce8ba
Oct 30 21:31:40.425401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:40.425367    4810 generic.go:356] "Generic (PLEG): container finished" podID="dd2835f0-fa2c-4a09-b169-5ca8c6a31b14" containerID="79af9bccb54f92d55ce6d8e0bd0799846701f790ea2543e1d9c40c5628f2397d" exitCode=0
Oct 30 21:31:40.425849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:40.425451    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb" event={"ID":"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14","Type":"ContainerDied","Data":"79af9bccb54f92d55ce6d8e0bd0799846701f790ea2543e1d9c40c5628f2397d"}
Oct 30 21:31:40.426824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:40.426796    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g" event={"ID":"54769587-dc35-44e4-9f79-7701a79f4aa0","Type":"ContainerStarted","Data":"ca1886235b9626c510ac70e091d6b5b9007f9435f9447dc87c87ab4ca6fce8ba"}
Oct 30 21:31:40.428466 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:40.428443    4810 generic.go:356] "Generic (PLEG): container finished" podID="b7b0e127-0fa4-436f-9599-8dc8081a3c62" containerID="47fa750a6948ad3a92d83e4d2d4fa3203c6cd7626baf8268211238bbc71a146d" exitCode=0
Oct 30 21:31:40.428565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:40.428471    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf" event={"ID":"b7b0e127-0fa4-436f-9599-8dc8081a3c62","Type":"ContainerDied","Data":"47fa750a6948ad3a92d83e4d2d4fa3203c6cd7626baf8268211238bbc71a146d"}
Oct 30 21:31:41.257281 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.257240    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6" podUID="ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:41.438418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.438381    4810 generic.go:356] "Generic (PLEG): container finished" podID="54769587-dc35-44e4-9f79-7701a79f4aa0" containerID="c59734059a6623e1465411fef34d389a594914c234cdd8542b3f4fc902478e4b" exitCode=0
Oct 30 21:31:41.438928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.438476    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g" event={"ID":"54769587-dc35-44e4-9f79-7701a79f4aa0","Type":"ContainerDied","Data":"c59734059a6623e1465411fef34d389a594914c234cdd8542b3f4fc902478e4b"}
Oct 30 21:31:41.698126 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.698050    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:41.700626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.700605    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:41.753827 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.753786    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-openshift-ca\") pod \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") "
Oct 30 21:31:41.754013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.753876    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-openshift-ca\") pod \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") "
Oct 30 21:31:41.754013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.753989    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-xks-ca\") pod \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") "
Oct 30 21:31:41.754159 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.754111    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b0e127-0fa4-436f-9599-8dc8081a3c62-kube-api-access-cfp9d\" (UniqueName: \"kubernetes.io/projected/b7b0e127-0fa4-436f-9599-8dc8081a3c62-kube-api-access-cfp9d\") pod \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") "
Oct 30 21:31:41.757897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.754226    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-kube-api-access-74cn4\" (UniqueName: \"kubernetes.io/projected/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-kube-api-access-74cn4\") pod \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\" (UID: \"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14\") "
Oct 30 21:31:41.757897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.754214    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-openshift-ca" pod "b7b0e127-0fa4-436f-9599-8dc8081a3c62" (UID: "b7b0e127-0fa4-436f-9599-8dc8081a3c62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:41.757897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.754281    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-xks-ca\") pod \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\" (UID: \"b7b0e127-0fa4-436f-9599-8dc8081a3c62\") "
Oct 30 21:31:41.757897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.754401    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-openshift-ca" pod "dd2835f0-fa2c-4a09-b169-5ca8c6a31b14" (UID: "dd2835f0-fa2c-4a09-b169-5ca8c6a31b14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:41.757897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.755114    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:41.757897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.755142    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:41.757897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.756610    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-xks-ca" pod "dd2835f0-fa2c-4a09-b169-5ca8c6a31b14" (UID: "dd2835f0-fa2c-4a09-b169-5ca8c6a31b14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:41.757897 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.756965    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-xks-ca" pod "b7b0e127-0fa4-436f-9599-8dc8081a3c62" (UID: "b7b0e127-0fa4-436f-9599-8dc8081a3c62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:41.758285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.757917    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-kube-api-access-74cn4" pod "dd2835f0-fa2c-4a09-b169-5ca8c6a31b14" (UID: "dd2835f0-fa2c-4a09-b169-5ca8c6a31b14"). InnerVolumeSpecName "kube-api-access-74cn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:41.758285 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.758143    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b0e127-0fa4-436f-9599-8dc8081a3c62-kube-api-access-cfp9d" pod "b7b0e127-0fa4-436f-9599-8dc8081a3c62" (UID: "b7b0e127-0fa4-436f-9599-8dc8081a3c62"). InnerVolumeSpecName "kube-api-access-cfp9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:41.856883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.856828    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:41.856883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.856887    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfp9d\" (UniqueName: \"kubernetes.io/projected/b7b0e127-0fa4-436f-9599-8dc8081a3c62-kube-api-access-cfp9d\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:41.857116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.856899    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74cn4\" (UniqueName: \"kubernetes.io/projected/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14-kube-api-access-74cn4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:41.857116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:41.856910    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7b0e127-0fa4-436f-9599-8dc8081a3c62-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:42.465332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.465285    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf" event={"ID":"b7b0e127-0fa4-436f-9599-8dc8081a3c62","Type":"ContainerDied","Data":"a44f423e73b66dcd040048d7ae307d9a0ff415ae0fe0cb332e93d6e6ef193505"}
Oct 30 21:31:42.465893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.465355    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"
Oct 30 21:31:42.465893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.465358    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a44f423e73b66dcd040048d7ae307d9a0ff415ae0fe0cb332e93d6e6ef193505"
Oct 30 21:31:42.467007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.466986    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"
Oct 30 21:31:42.467123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.467017    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb" event={"ID":"dd2835f0-fa2c-4a09-b169-5ca8c6a31b14","Type":"ContainerDied","Data":"aee71f7555239d48246d80b3e7f923da4832a15cf2c246580ca127fafc22d18e"}
Oct 30 21:31:42.467123 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.467058    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee71f7555239d48246d80b3e7f923da4832a15cf2c246580ca127fafc22d18e"
Oct 30 21:31:42.545601 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:42.545567    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b0e127_0fa4_436f_9599_8dc8081a3c62.slice/crio-a44f423e73b66dcd040048d7ae307d9a0ff415ae0fe0cb332e93d6e6ef193505\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b0e127_0fa4_436f_9599_8dc8081a3c62.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:31:42.653097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.653068    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:42.767829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.767731    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-openshift-ca\") pod \"54769587-dc35-44e4-9f79-7701a79f4aa0\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") "
Oct 30 21:31:42.767829 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.767820    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-xks-ca\") pod \"54769587-dc35-44e4-9f79-7701a79f4aa0\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") "
Oct 30 21:31:42.768120 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.768035    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54769587-dc35-44e4-9f79-7701a79f4aa0-kube-api-access-9npwb\" (UniqueName: \"kubernetes.io/projected/54769587-dc35-44e4-9f79-7701a79f4aa0-kube-api-access-9npwb\") pod \"54769587-dc35-44e4-9f79-7701a79f4aa0\" (UID: \"54769587-dc35-44e4-9f79-7701a79f4aa0\") "
Oct 30 21:31:42.768251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.768227    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-openshift-ca" pod "54769587-dc35-44e4-9f79-7701a79f4aa0" (UID: "54769587-dc35-44e4-9f79-7701a79f4aa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:42.768738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.768713    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:42.770316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.770289    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54769587-dc35-44e4-9f79-7701a79f4aa0-kube-api-access-9npwb" pod "54769587-dc35-44e4-9f79-7701a79f4aa0" (UID: "54769587-dc35-44e4-9f79-7701a79f4aa0"). InnerVolumeSpecName "kube-api-access-9npwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:42.770447 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.770299    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-xks-ca" pod "54769587-dc35-44e4-9f79-7701a79f4aa0" (UID: "54769587-dc35-44e4-9f79-7701a79f4aa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:42.869842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.869805    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9npwb\" (UniqueName: \"kubernetes.io/projected/54769587-dc35-44e4-9f79-7701a79f4aa0-kube-api-access-9npwb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:42.869842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:42.869835    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54769587-dc35-44e4-9f79-7701a79f4aa0-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:43.487489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:43.487454    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g"
Oct 30 21:31:43.488031 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:43.487504    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07a74d0-q2c4g" event={"ID":"54769587-dc35-44e4-9f79-7701a79f4aa0","Type":"ContainerDied","Data":"ca1886235b9626c510ac70e091d6b5b9007f9435f9447dc87c87ab4ca6fce8ba"}
Oct 30 21:31:43.488031 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:43.487569    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1886235b9626c510ac70e091d6b5b9007f9435f9447dc87c87ab4ca6fce8ba"
Oct 30 21:31:45.980500 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:45.980474    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:31:46.016683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.016658    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:31:46.121717 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.121684    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-plugins-config-volume\") pod \"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e\" (UID: \"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e\") "
Oct 30 21:31:46.121972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.121761    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19991362-bc52-4564-b54e-64f087a20820-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19991362-bc52-4564-b54e-64f087a20820-plugins-config-volume\") pod \"19991362-bc52-4564-b54e-64f087a20820\" (UID: \"19991362-bc52-4564-b54e-64f087a20820\") "
Oct 30 21:31:46.121972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.121849    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19991362-bc52-4564-b54e-64f087a20820-kube-api-access-c6n9b\" (UniqueName: \"kubernetes.io/projected/19991362-bc52-4564-b54e-64f087a20820-kube-api-access-c6n9b\") pod \"19991362-bc52-4564-b54e-64f087a20820\" (UID: \"19991362-bc52-4564-b54e-64f087a20820\") "
Oct 30 21:31:46.121972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.121915    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-kube-api-access-fghxw\" (UniqueName: \"kubernetes.io/projected/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-kube-api-access-fghxw\") pod \"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e\" (UID: \"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e\") "
Oct 30 21:31:46.122218 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.122190    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-plugins-config-volume" pod "366d6f28-7835-4ea2-9ebf-32bfa47f3e0e" (UID: "366d6f28-7835-4ea2-9ebf-32bfa47f3e0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:46.122301 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.122235    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19991362-bc52-4564-b54e-64f087a20820-plugins-config-volume" pod "19991362-bc52-4564-b54e-64f087a20820" (UID: "19991362-bc52-4564-b54e-64f087a20820"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:46.122448 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.122429    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:46.122536 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.122449    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19991362-bc52-4564-b54e-64f087a20820-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:46.124066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.124040    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19991362-bc52-4564-b54e-64f087a20820-kube-api-access-c6n9b" pod "19991362-bc52-4564-b54e-64f087a20820" (UID: "19991362-bc52-4564-b54e-64f087a20820"). InnerVolumeSpecName "kube-api-access-c6n9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:46.124170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.124144    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-kube-api-access-fghxw" pod "366d6f28-7835-4ea2-9ebf-32bfa47f3e0e" (UID: "366d6f28-7835-4ea2-9ebf-32bfa47f3e0e"). InnerVolumeSpecName "kube-api-access-fghxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:46.223743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.223649    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6n9b\" (UniqueName: \"kubernetes.io/projected/19991362-bc52-4564-b54e-64f087a20820-kube-api-access-c6n9b\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:46.223743 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.223687    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fghxw\" (UniqueName: \"kubernetes.io/projected/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e-kube-api-access-fghxw\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:46.514524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.514413    4810 generic.go:356] "Generic (PLEG): container finished" podID="19991362-bc52-4564-b54e-64f087a20820" containerID="b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6" exitCode=0
Oct 30 21:31:46.514524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.514460    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"
Oct 30 21:31:46.514524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.514502    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c" event={"ID":"19991362-bc52-4564-b54e-64f087a20820","Type":"ContainerDied","Data":"b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6"}
Oct 30 21:31:46.514796 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.514548    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c" event={"ID":"19991362-bc52-4564-b54e-64f087a20820","Type":"ContainerDied","Data":"ddd21b5749100932d7097518f71581c1bc9bc03b1fa141dc244a534d96785157"}
Oct 30 21:31:46.514796 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.514576    4810 scope.go:122] "RemoveContainer" containerID="b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6"
Oct 30 21:31:46.516171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.516147    4810 generic.go:356] "Generic (PLEG): container finished" podID="366d6f28-7835-4ea2-9ebf-32bfa47f3e0e" containerID="86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7" exitCode=0
Oct 30 21:31:46.516374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.516357    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"
Oct 30 21:31:46.516446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.516374    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7" event={"ID":"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e","Type":"ContainerDied","Data":"86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7"}
Oct 30 21:31:46.516446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.516414    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7" event={"ID":"366d6f28-7835-4ea2-9ebf-32bfa47f3e0e","Type":"ContainerDied","Data":"49155a591aab653276fb0f84869e1ba5626b381d526649dfae1a2808330dbd6e"}
Oct 30 21:31:46.527519 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.527495    4810 scope.go:122] "RemoveContainer" containerID="b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6"
Oct 30 21:31:46.527900 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:46.527846    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6\": container with ID starting with b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6 not found: ID does not exist" containerID="b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6"
Oct 30 21:31:46.527979 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.527910    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6"} err="failed to get container status \"b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6\": rpc error: code = NotFound desc = could not find container \"b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6\": container with ID starting with b545c15c117d440daebab1c2a79d687521b523d524c830a4af04a3c4261bd2a6 not found: ID does not exist"
Oct 30 21:31:46.527979 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.527929    4810 scope.go:122] "RemoveContainer" containerID="86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7"
Oct 30 21:31:46.539381 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.539355    4810 scope.go:122] "RemoveContainer" containerID="86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7"
Oct 30 21:31:46.539711 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:46.539680    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7\": container with ID starting with 86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7 not found: ID does not exist" containerID="86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7"
Oct 30 21:31:46.539892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.539714    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7"} err="failed to get container status \"86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7\": rpc error: code = NotFound desc = could not find container \"86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7\": container with ID starting with 86e45da45e2d61b850e519da3ce07069601ce77e6ad7e1ff228e271994a6d3a7 not found: ID does not exist"
Oct 30 21:31:46.544094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.544070    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"]
Oct 30 21:31:46.567745 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.567705    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e1e0f5-9c9d969cc-ztr4c"]
Oct 30 21:31:46.571237 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.571207    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"]
Oct 30 21:31:46.574359 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:46.574335    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e1e0f5-5dd7c6fff-twpp7"]
Oct 30 21:31:47.667910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:47.667872    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19991362-bc52-4564-b54e-64f087a20820" path="/var/lib/kubelet/pods/19991362-bc52-4564-b54e-64f087a20820/volumes"
Oct 30 21:31:47.669431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:47.669401    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366d6f28-7835-4ea2-9ebf-32bfa47f3e0e" path="/var/lib/kubelet/pods/366d6f28-7835-4ea2-9ebf-32bfa47f3e0e/volumes"
Oct 30 21:31:47.833539 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:47.833503    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"]
Oct 30 21:31:47.836767 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:47.836736    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3a65ec-jbsgn"]
Oct 30 21:31:48.459326 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:48.459292    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"]
Oct 30 21:31:48.738308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:48.738226    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"]
Oct 30 21:31:49.257730 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:49.257693    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8" podUID="b258a27c-5663-494e-abff-b1979edfee61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:31:49.670666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:49.670630    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5" path="/var/lib/kubelet/pods/24ca0920-2ce3-4b4b-a082-cc4ff3cb40c5/volumes"
Oct 30 21:31:49.753261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:49.753236    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:31:49.868865 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:49.868776    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b258a27c-5663-494e-abff-b1979edfee61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b258a27c-5663-494e-abff-b1979edfee61-plugins-config-volume\") pod \"b258a27c-5663-494e-abff-b1979edfee61\" (UID: \"b258a27c-5663-494e-abff-b1979edfee61\") "
Oct 30 21:31:49.869036 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:49.868854    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b258a27c-5663-494e-abff-b1979edfee61-kube-api-access-k75wn\" (UniqueName: \"kubernetes.io/projected/b258a27c-5663-494e-abff-b1979edfee61-kube-api-access-k75wn\") pod \"b258a27c-5663-494e-abff-b1979edfee61\" (UID: \"b258a27c-5663-494e-abff-b1979edfee61\") "
Oct 30 21:31:49.869214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:49.869192    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b258a27c-5663-494e-abff-b1979edfee61-plugins-config-volume" pod "b258a27c-5663-494e-abff-b1979edfee61" (UID: "b258a27c-5663-494e-abff-b1979edfee61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:49.869316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:49.869302    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b258a27c-5663-494e-abff-b1979edfee61-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:49.871150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:49.871120    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b258a27c-5663-494e-abff-b1979edfee61-kube-api-access-k75wn" pod "b258a27c-5663-494e-abff-b1979edfee61" (UID: "b258a27c-5663-494e-abff-b1979edfee61"). InnerVolumeSpecName "kube-api-access-k75wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:49.970428 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:49.970389    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k75wn\" (UniqueName: \"kubernetes.io/projected/b258a27c-5663-494e-abff-b1979edfee61-kube-api-access-k75wn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:50.239513 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.239473    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"]
Oct 30 21:31:50.447370 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.447333    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"]
Oct 30 21:31:50.448148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448125    4810 cpu_manager.go:414] "containerMap: removing container" podUID="19991362-bc52-4564-b54e-64f087a20820" containerName="payload-processing"
Oct 30 21:31:50.448217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448150    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19991362-bc52-4564-b54e-64f087a20820" containerName="payload-processing"
Oct 30 21:31:50.448217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448168    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dd2835f0-fa2c-4a09-b169-5ca8c6a31b14" containerName="revoke-keys"
Oct 30 21:31:50.448217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448173    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2835f0-fa2c-4a09-b169-5ca8c6a31b14" containerName="revoke-keys"
Oct 30 21:31:50.448217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448186    4810 cpu_manager.go:414] "containerMap: removing container" podUID="b258a27c-5663-494e-abff-b1979edfee61" containerName="payload-processing"
Oct 30 21:31:50.448217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448191    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b258a27c-5663-494e-abff-b1979edfee61" containerName="payload-processing"
Oct 30 21:31:50.448217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448201    4810 cpu_manager.go:414] "containerMap: removing container" podUID="54769587-dc35-44e4-9f79-7701a79f4aa0" containerName="revoke-keys"
Oct 30 21:31:50.448217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448205    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54769587-dc35-44e4-9f79-7701a79f4aa0" containerName="revoke-keys"
Oct 30 21:31:50.448450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448227    4810 cpu_manager.go:414] "containerMap: removing container" podUID="366d6f28-7835-4ea2-9ebf-32bfa47f3e0e" containerName="payload-pre-processing"
Oct 30 21:31:50.448450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448232    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366d6f28-7835-4ea2-9ebf-32bfa47f3e0e" containerName="payload-pre-processing"
Oct 30 21:31:50.448450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448240    4810 cpu_manager.go:414] "containerMap: removing container" podUID="b7b0e127-0fa4-436f-9599-8dc8081a3c62" containerName="revoke-keys"
Oct 30 21:31:50.448450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448246    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b0e127-0fa4-436f-9599-8dc8081a3c62" containerName="revoke-keys"
Oct 30 21:31:50.448450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448335    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="b258a27c-5663-494e-abff-b1979edfee61" containerName="payload-processing"
Oct 30 21:31:50.448450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448344    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="54769587-dc35-44e4-9f79-7701a79f4aa0" containerName="revoke-keys"
Oct 30 21:31:50.448450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448355    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="366d6f28-7835-4ea2-9ebf-32bfa47f3e0e" containerName="payload-pre-processing"
Oct 30 21:31:50.448450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448361    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b0e127-0fa4-436f-9599-8dc8081a3c62" containerName="revoke-keys"
Oct 30 21:31:50.448450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448370    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="19991362-bc52-4564-b54e-64f087a20820" containerName="payload-processing"
Oct 30 21:31:50.448450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.448378    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2835f0-fa2c-4a09-b169-5ca8c6a31b14" containerName="revoke-keys"
Oct 30 21:31:50.451953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.451933    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.459886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.457114    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:31:50.459886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.457174    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:50.461479 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.461423    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"]
Oct 30 21:31:50.566189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.566156    4810 generic.go:356] "Generic (PLEG): container finished" podID="b258a27c-5663-494e-abff-b1979edfee61" containerID="bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796" exitCode=0
Oct 30 21:31:50.566402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.566251    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8" event={"ID":"b258a27c-5663-494e-abff-b1979edfee61","Type":"ContainerDied","Data":"bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796"}
Oct 30 21:31:50.566402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.566302    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8" event={"ID":"b258a27c-5663-494e-abff-b1979edfee61","Type":"ContainerDied","Data":"4460f72d098b0aac39a90794481e3f48df9ad8da6ce72f62751bea6da90ed566"}
Oct 30 21:31:50.566402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.566331    4810 scope.go:122] "RemoveContainer" containerID="bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796"
Oct 30 21:31:50.566584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.566476    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"
Oct 30 21:31:50.569821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.569784    4810 generic.go:356] "Generic (PLEG): container finished" podID="ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1" containerID="542c215b15de66e055e780fbc4d641ac62c45950d66c7ac92871400bd0221fa7" exitCode=0
Oct 30 21:31:50.570051 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.569998    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6" event={"ID":"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1","Type":"ContainerDied","Data":"542c215b15de66e055e780fbc4d641ac62c45950d66c7ac92871400bd0221fa7"}
Oct 30 21:31:50.587686 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.587657    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnsqp\" (UniqueName: \"kubernetes.io/projected/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-kube-api-access-xnsqp\") pod \"maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.587924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.587826    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.587924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.587914    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.591028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.591007    4810 scope.go:122] "RemoveContainer" containerID="bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796"
Oct 30 21:31:50.591759 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:50.591506    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796\": container with ID starting with bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796 not found: ID does not exist" containerID="bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796"
Oct 30 21:31:50.591759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.591548    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796"} err="failed to get container status \"bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796\": rpc error: code = NotFound desc = could not find container \"bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796\": container with ID starting with bbd7d9872a2301e96be537344225d88d0b33da012433f1f086f54264c5a62796 not found: ID does not exist"
Oct 30 21:31:50.617672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.617644    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"]
Oct 30 21:31:50.625074 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.624319    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e07442-8f78fbd57-w9ql8"]
Oct 30 21:31:50.689910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.689832    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.690679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.689927    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.690679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.690026    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnsqp\" (UniqueName: \"kubernetes.io/projected/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-kube-api-access-xnsqp\") pod \"maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.691271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.691244    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.704003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.703934    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.706200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.706154    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnsqp\" (UniqueName: \"kubernetes.io/projected/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-kube-api-access-xnsqp\") pod \"maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.742905 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.742870    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:31:50.813956 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.813737    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:50.899880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.894319    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-kube-api-access-9p2ct\" (UniqueName: \"kubernetes.io/projected/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-kube-api-access-9p2ct\") pod \"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1\" (UID: \"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1\") "
Oct 30 21:31:50.899880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.894532    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-plugins-config-volume\") pod \"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1\" (UID: \"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1\") "
Oct 30 21:31:50.904878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.902127    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-plugins-config-volume" pod "ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1" (UID: "ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:50.913529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.911039    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-kube-api-access-9p2ct" pod "ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1" (UID: "ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1"). InnerVolumeSpecName "kube-api-access-9p2ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:50.997117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.996698    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:50.997117 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:50.996728    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p2ct\" (UniqueName: \"kubernetes.io/projected/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1-kube-api-access-9p2ct\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:51.158102 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.158077    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"]
Oct 30 21:31:51.159060 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:31:51.159033    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e8e825_6cb3_4dd8_95ab_274ed6dbd62c.slice/crio-317da1d8d079061f034efacf6d556282cfbf2dee40600c1f39856f20a02322b0 WatchSource:0}: Error finding container 317da1d8d079061f034efacf6d556282cfbf2dee40600c1f39856f20a02322b0: Status 404 returned error can't find the container with id 317da1d8d079061f034efacf6d556282cfbf2dee40600c1f39856f20a02322b0
Oct 30 21:31:51.583773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.583737    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"
Oct 30 21:31:51.584013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.583736    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6" event={"ID":"ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1","Type":"ContainerDied","Data":"15938a39bff74c3f598ae1f29090c8c2dc92e575f01f908a4e5f898b13d98c2a"}
Oct 30 21:31:51.584013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.583926    4810 scope.go:122] "RemoveContainer" containerID="542c215b15de66e055e780fbc4d641ac62c45950d66c7ac92871400bd0221fa7"
Oct 30 21:31:51.586691 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.586653    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf" event={"ID":"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c","Type":"ContainerStarted","Data":"3b80ce72a3e5f309f6e67c6a42ec225be71f68cef9e0ed3c1a5c2eb5978b4d5f"}
Oct 30 21:31:51.586837 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.586707    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf" event={"ID":"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c","Type":"ContainerStarted","Data":"317da1d8d079061f034efacf6d556282cfbf2dee40600c1f39856f20a02322b0"}
Oct 30 21:31:51.607841 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.607758    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf" podStartSLOduration=1.607745201 podStartE2EDuration="1.607745201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:51.60665669 +0000 UTC m=+3278.098473235" watchObservedRunningTime="2026-10-30 21:31:51.607745201 +0000 UTC m=+3278.099561745"
Oct 30 21:31:51.624627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.624599    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"]
Oct 30 21:31:51.632085 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.632051    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-00909f-6c5f4bff55-5x7r6"]
Oct 30 21:31:51.662690 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.662661    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b258a27c-5663-494e-abff-b1979edfee61" path="/var/lib/kubelet/pods/b258a27c-5663-494e-abff-b1979edfee61/volumes"
Oct 30 21:31:51.663132 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:51.663118    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1" path="/var/lib/kubelet/pods/ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1/volumes"
Oct 30 21:31:52.600372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:52.600334    4810 generic.go:356] "Generic (PLEG): container finished" podID="85e8e825-6cb3-4dd8-95ab-274ed6dbd62c" containerID="3b80ce72a3e5f309f6e67c6a42ec225be71f68cef9e0ed3c1a5c2eb5978b4d5f" exitCode=0
Oct 30 21:31:52.600903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:52.600383    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf" event={"ID":"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c","Type":"ContainerDied","Data":"3b80ce72a3e5f309f6e67c6a42ec225be71f68cef9e0ed3c1a5c2eb5978b4d5f"}
Oct 30 21:31:53.460064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.459998    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w" podUID="5491ea4b-00ab-4989-8198-33d1f26b5a8f" containerName="payload-pre-processing" containerID="cri-o://c7ba3199280d4186a957d84f953f418a73576d8481cc4ae9c1340a7507103d4e" gracePeriod=25
Oct 30 21:31:53.610786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.610742    4810 generic.go:356] "Generic (PLEG): container finished" podID="5491ea4b-00ab-4989-8198-33d1f26b5a8f" containerID="c7ba3199280d4186a957d84f953f418a73576d8481cc4ae9c1340a7507103d4e" exitCode=0
Oct 30 21:31:53.611231 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.610828    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w" event={"ID":"5491ea4b-00ab-4989-8198-33d1f26b5a8f","Type":"ContainerDied","Data":"c7ba3199280d4186a957d84f953f418a73576d8481cc4ae9c1340a7507103d4e"}
Oct 30 21:31:53.739414 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.739335    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd" podUID="9846a273-e5af-4bfc-b365-0d3fb804e5c4" containerName="payload-processing" containerID="cri-o://8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe" gracePeriod=25
Oct 30 21:31:53.774456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.774421    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:31:53.821341 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.821312    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:53.938112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.938080    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5491ea4b-00ab-4989-8198-33d1f26b5a8f-kube-api-access-7txmn\" (UniqueName: \"kubernetes.io/projected/5491ea4b-00ab-4989-8198-33d1f26b5a8f-kube-api-access-7txmn\") pod \"5491ea4b-00ab-4989-8198-33d1f26b5a8f\" (UID: \"5491ea4b-00ab-4989-8198-33d1f26b5a8f\") "
Oct 30 21:31:53.938308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.938151    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-openshift-ca\") pod \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") "
Oct 30 21:31:53.938308 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.938298    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-xks-ca\") pod \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") "
Oct 30 21:31:53.938456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.938384    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-kube-api-access-xnsqp\" (UniqueName: \"kubernetes.io/projected/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-kube-api-access-xnsqp\") pod \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\" (UID: \"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c\") "
Oct 30 21:31:53.938456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.938424    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5491ea4b-00ab-4989-8198-33d1f26b5a8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5491ea4b-00ab-4989-8198-33d1f26b5a8f-plugins-config-volume\") pod \"5491ea4b-00ab-4989-8198-33d1f26b5a8f\" (UID: \"5491ea4b-00ab-4989-8198-33d1f26b5a8f\") "
Oct 30 21:31:53.938560 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.938497    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-openshift-ca" pod "85e8e825-6cb3-4dd8-95ab-274ed6dbd62c" (UID: "85e8e825-6cb3-4dd8-95ab-274ed6dbd62c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:53.938899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.938840    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5491ea4b-00ab-4989-8198-33d1f26b5a8f-plugins-config-volume" pod "5491ea4b-00ab-4989-8198-33d1f26b5a8f" (UID: "5491ea4b-00ab-4989-8198-33d1f26b5a8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:53.939053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.939023    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5491ea4b-00ab-4989-8198-33d1f26b5a8f-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:53.939053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.939046    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:53.940563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.940541    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-xks-ca" pod "85e8e825-6cb3-4dd8-95ab-274ed6dbd62c" (UID: "85e8e825-6cb3-4dd8-95ab-274ed6dbd62c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:53.940663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.940561    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-kube-api-access-xnsqp" pod "85e8e825-6cb3-4dd8-95ab-274ed6dbd62c" (UID: "85e8e825-6cb3-4dd8-95ab-274ed6dbd62c"). InnerVolumeSpecName "kube-api-access-xnsqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:53.940663 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:53.940589    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5491ea4b-00ab-4989-8198-33d1f26b5a8f-kube-api-access-7txmn" pod "5491ea4b-00ab-4989-8198-33d1f26b5a8f" (UID: "5491ea4b-00ab-4989-8198-33d1f26b5a8f"). InnerVolumeSpecName "kube-api-access-7txmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:54.040629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.040544    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7txmn\" (UniqueName: \"kubernetes.io/projected/5491ea4b-00ab-4989-8198-33d1f26b5a8f-kube-api-access-7txmn\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:54.040629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.040576    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:54.040629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.040587    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnsqp\" (UniqueName: \"kubernetes.io/projected/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c-kube-api-access-xnsqp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:54.166723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.166678    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm" podUID="87d65735-3af5-46d9-9bd5-b616c14ab940" containerName="main" containerID="cri-o://b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c" gracePeriod=45
Oct 30 21:31:54.428210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.428186    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:31:54.551039 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.550979    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-home\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-home\") pod \"87d65735-3af5-46d9-9bd5-b616c14ab940\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") "
Oct 30 21:31:54.551292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.551079    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d65735-3af5-46d9-9bd5-b616c14ab940-kube-api-access-5nkj5\" (UniqueName: \"kubernetes.io/projected/87d65735-3af5-46d9-9bd5-b616c14ab940-kube-api-access-5nkj5\") pod \"87d65735-3af5-46d9-9bd5-b616c14ab940\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") "
Oct 30 21:31:54.551292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.551114    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-dshm\") pod \"87d65735-3af5-46d9-9bd5-b616c14ab940\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") "
Oct 30 21:31:54.551292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.551247    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87d65735-3af5-46d9-9bd5-b616c14ab940-tls-certs\" (UniqueName: \"kubernetes.io/secret/87d65735-3af5-46d9-9bd5-b616c14ab940-tls-certs\") pod \"87d65735-3af5-46d9-9bd5-b616c14ab940\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") "
Oct 30 21:31:54.551495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.551295    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-tmp-dir\") pod \"87d65735-3af5-46d9-9bd5-b616c14ab940\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") "
Oct 30 21:31:54.551495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.551321    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-home" pod "87d65735-3af5-46d9-9bd5-b616c14ab940" (UID: "87d65735-3af5-46d9-9bd5-b616c14ab940"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:54.551639 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.551494    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-model-cache\") pod \"87d65735-3af5-46d9-9bd5-b616c14ab940\" (UID: \"87d65735-3af5-46d9-9bd5-b616c14ab940\") "
Oct 30 21:31:54.551723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.551666    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-tmp-dir" pod "87d65735-3af5-46d9-9bd5-b616c14ab940" (UID: "87d65735-3af5-46d9-9bd5-b616c14ab940"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:54.552009 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.551980    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-model-cache" pod "87d65735-3af5-46d9-9bd5-b616c14ab940" (UID: "87d65735-3af5-46d9-9bd5-b616c14ab940"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:54.552306 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.552287    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:54.552376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.552312    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:54.552376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.552328    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:54.553882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.553834    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-dshm" pod "87d65735-3af5-46d9-9bd5-b616c14ab940" (UID: "87d65735-3af5-46d9-9bd5-b616c14ab940"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:54.553970 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.553910    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d65735-3af5-46d9-9bd5-b616c14ab940-kube-api-access-5nkj5" pod "87d65735-3af5-46d9-9bd5-b616c14ab940" (UID: "87d65735-3af5-46d9-9bd5-b616c14ab940"). InnerVolumeSpecName "kube-api-access-5nkj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:54.554230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.554207    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d65735-3af5-46d9-9bd5-b616c14ab940-tls-certs" pod "87d65735-3af5-46d9-9bd5-b616c14ab940" (UID: "87d65735-3af5-46d9-9bd5-b616c14ab940"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:54.620324 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.620248    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"
Oct 30 21:31:54.620728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.620245    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf" event={"ID":"85e8e825-6cb3-4dd8-95ab-274ed6dbd62c","Type":"ContainerDied","Data":"317da1d8d079061f034efacf6d556282cfbf2dee40600c1f39856f20a02322b0"}
Oct 30 21:31:54.620728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.620397    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317da1d8d079061f034efacf6d556282cfbf2dee40600c1f39856f20a02322b0"
Oct 30 21:31:54.621936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.621916    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"
Oct 30 21:31:54.622058 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.621928    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w" event={"ID":"5491ea4b-00ab-4989-8198-33d1f26b5a8f","Type":"ContainerDied","Data":"fdd529027379f0dc78f13489e956b35363fcb0feab75c9396e23eeedaa54c210"}
Oct 30 21:31:54.622058 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.621982    4810 scope.go:122] "RemoveContainer" containerID="c7ba3199280d4186a957d84f953f418a73576d8481cc4ae9c1340a7507103d4e"
Oct 30 21:31:54.623705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.623671    4810 generic.go:356] "Generic (PLEG): container finished" podID="87d65735-3af5-46d9-9bd5-b616c14ab940" containerID="b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c" exitCode=0
Oct 30 21:31:54.623844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.623698    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm" event={"ID":"87d65735-3af5-46d9-9bd5-b616c14ab940","Type":"ContainerDied","Data":"b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c"}
Oct 30 21:31:54.623844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.623717    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"
Oct 30 21:31:54.623844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.623739    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm" event={"ID":"87d65735-3af5-46d9-9bd5-b616c14ab940","Type":"ContainerDied","Data":"868a8ebfbbce424e9f58a25ce895b8eb7da6df4049dd918e5e04e080fff8d7eb"}
Oct 30 21:31:54.637100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.637078    4810 scope.go:122] "RemoveContainer" containerID="b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c"
Oct 30 21:31:54.653456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.653350    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d65735-3af5-46d9-9bd5-b616c14ab940-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:54.653456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.653374    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkj5\" (UniqueName: \"kubernetes.io/projected/87d65735-3af5-46d9-9bd5-b616c14ab940-kube-api-access-5nkj5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:54.653456 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.653390    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d65735-3af5-46d9-9bd5-b616c14ab940-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:54.653620 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.653517    4810 scope.go:122] "RemoveContainer" containerID="b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c"
Oct 30 21:31:54.653893 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:54.653850    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c\": container with ID starting with b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c not found: ID does not exist" containerID="b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c"
Oct 30 21:31:54.653984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.653903    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c"} err="failed to get container status \"b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c\": rpc error: code = NotFound desc = could not find container \"b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c\": container with ID starting with b8a11c3450880f25fb84cdf9f0c2fcd32075afec03384eb1b1cf485248cb173c not found: ID does not exist"
Oct 30 21:31:54.682527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.682466    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"]
Oct 30 21:31:54.721946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.721911    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f147de-7c6dcb6dff-xj76w"]
Oct 30 21:31:54.751516 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.751480    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"]
Oct 30 21:31:54.785450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:54.785419    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0c4052/test-model-0c4052-kserve-6bd47968dc-7bgrm"]
Oct 30 21:31:55.167608 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:31:55.167550    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:31:55.663083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:55.663055    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5491ea4b-00ab-4989-8198-33d1f26b5a8f" path="/var/lib/kubelet/pods/5491ea4b-00ab-4989-8198-33d1f26b5a8f/volumes"
Oct 30 21:31:55.663516 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:55.663502    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d65735-3af5-46d9-9bd5-b616c14ab940" path="/var/lib/kubelet/pods/87d65735-3af5-46d9-9bd5-b616c14ab940/volumes"
Oct 30 21:31:56.304049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304010    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"]
Oct 30 21:31:56.304644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304627    4810 cpu_manager.go:414] "containerMap: removing container" podUID="87d65735-3af5-46d9-9bd5-b616c14ab940" containerName="main"
Oct 30 21:31:56.304703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304647    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d65735-3af5-46d9-9bd5-b616c14ab940" containerName="main"
Oct 30 21:31:56.304703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304660    4810 cpu_manager.go:414] "containerMap: removing container" podUID="85e8e825-6cb3-4dd8-95ab-274ed6dbd62c" containerName="revoke-keys"
Oct 30 21:31:56.304703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304666    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e8e825-6cb3-4dd8-95ab-274ed6dbd62c" containerName="revoke-keys"
Oct 30 21:31:56.304703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304680    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1" containerName="payload-pre-processing"
Oct 30 21:31:56.304703 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304685    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1" containerName="payload-pre-processing"
Oct 30 21:31:56.304922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304704    4810 cpu_manager.go:414] "containerMap: removing container" podUID="5491ea4b-00ab-4989-8198-33d1f26b5a8f" containerName="payload-pre-processing"
Oct 30 21:31:56.304922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304712    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5491ea4b-00ab-4989-8198-33d1f26b5a8f" containerName="payload-pre-processing"
Oct 30 21:31:56.304922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304833    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="5491ea4b-00ab-4989-8198-33d1f26b5a8f" containerName="payload-pre-processing"
Oct 30 21:31:56.304922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304849    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d65735-3af5-46d9-9bd5-b616c14ab940" containerName="main"
Oct 30 21:31:56.304922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304870    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e8e825-6cb3-4dd8-95ab-274ed6dbd62c" containerName="revoke-keys"
Oct 30 21:31:56.304922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.304880    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0d6e1c-9bc7-4a21-9e93-aeafa944b6a1" containerName="payload-pre-processing"
Oct 30 21:31:56.309584 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.309568    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.316452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.316428    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"]
Oct 30 21:31:56.316759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.316735    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:56.316974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.316950    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:31:56.477646 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.477611    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq4vr\" (UniqueName: \"kubernetes.io/projected/d909606f-340f-471b-b7d3-129f80e23881-kube-api-access-hq4vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.477853 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.477743    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d909606f-340f-471b-b7d3-129f80e23881-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.477853 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.477847    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d909606f-340f-471b-b7d3-129f80e23881-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.585662 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.579458    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d909606f-340f-471b-b7d3-129f80e23881-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.585662 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.579592    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq4vr\" (UniqueName: \"kubernetes.io/projected/d909606f-340f-471b-b7d3-129f80e23881-kube-api-access-hq4vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.585662 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.579683    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d909606f-340f-471b-b7d3-129f80e23881-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.585662 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.580902    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d909606f-340f-471b-b7d3-129f80e23881-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.585662 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.585401    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d909606f-340f-471b-b7d3-129f80e23881-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.603302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.603273    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq4vr\" (UniqueName: \"kubernetes.io/projected/d909606f-340f-471b-b7d3-129f80e23881-kube-api-access-hq4vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.635796 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.635759    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:56.942842 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:56.942808    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"]
Oct 30 21:31:56.947965 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:31:56.947930    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd909606f_340f_471b_b7d3_129f80e23881.slice/crio-42fa3c5b6fba1e482f869b3e05cd25f04e6c51acc3bac1dd540b807e1f4a21ba WatchSource:0}: Error finding container 42fa3c5b6fba1e482f869b3e05cd25f04e6c51acc3bac1dd540b807e1f4a21ba: Status 404 returned error can't find the container with id 42fa3c5b6fba1e482f869b3e05cd25f04e6c51acc3bac1dd540b807e1f4a21ba
Oct 30 21:31:57.092706 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:57.092662    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"]
Oct 30 21:31:57.095477 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:57.095452    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8fa15b-2xghf"]
Oct 30 21:31:57.663928 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:57.663897    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7b0e127-0fa4-436f-9599-8dc8081a3c62" path="/var/lib/kubelet/pods/b7b0e127-0fa4-436f-9599-8dc8081a3c62/volumes"
Oct 30 21:31:57.664242 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:57.664224    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44" event={"ID":"d909606f-340f-471b-b7d3-129f80e23881","Type":"ContainerStarted","Data":"7797da01444b48a6f8818a308d0cbd38ebefd09f8fdb0ab444e51efba328b81c"}
Oct 30 21:31:57.664302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:57.664264    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44" event={"ID":"d909606f-340f-471b-b7d3-129f80e23881","Type":"ContainerStarted","Data":"42fa3c5b6fba1e482f869b3e05cd25f04e6c51acc3bac1dd540b807e1f4a21ba"}
Oct 30 21:31:57.690734 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:57.690686    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44" podStartSLOduration=1.6906732679999998 podStartE2EDuration="1.690673268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:57.687906461 +0000 UTC m=+3284.179723006" watchObservedRunningTime="2026-10-30 21:31:57.690673268 +0000 UTC m=+3284.182489813"
Oct 30 21:31:58.665214 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:58.665175    4810 generic.go:356] "Generic (PLEG): container finished" podID="d909606f-340f-471b-b7d3-129f80e23881" containerID="7797da01444b48a6f8818a308d0cbd38ebefd09f8fdb0ab444e51efba328b81c" exitCode=0
Oct 30 21:31:58.665607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:58.665237    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44" event={"ID":"d909606f-340f-471b-b7d3-129f80e23881","Type":"ContainerDied","Data":"7797da01444b48a6f8818a308d0cbd38ebefd09f8fdb0ab444e51efba328b81c"}
Oct 30 21:31:59.104924 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:59.104826    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"]
Oct 30 21:31:59.109333 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:59.109296    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f147de-xstsb"]
Oct 30 21:31:59.665726 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:59.665685    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2835f0-fa2c-4a09-b169-5ca8c6a31b14" path="/var/lib/kubelet/pods/dd2835f0-fa2c-4a09-b169-5ca8c6a31b14/volumes"
Oct 30 21:31:59.829046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:59.829013    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:31:59.930400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:59.930332    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d909606f-340f-471b-b7d3-129f80e23881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d909606f-340f-471b-b7d3-129f80e23881-maas-api-xks-ca\") pod \"d909606f-340f-471b-b7d3-129f80e23881\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") "
Oct 30 21:31:59.930565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:59.930515    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d909606f-340f-471b-b7d3-129f80e23881-kube-api-access-hq4vr\" (UniqueName: \"kubernetes.io/projected/d909606f-340f-471b-b7d3-129f80e23881-kube-api-access-hq4vr\") pod \"d909606f-340f-471b-b7d3-129f80e23881\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") "
Oct 30 21:31:59.930565 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:59.930550    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d909606f-340f-471b-b7d3-129f80e23881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d909606f-340f-471b-b7d3-129f80e23881-maas-api-openshift-ca\") pod \"d909606f-340f-471b-b7d3-129f80e23881\" (UID: \"d909606f-340f-471b-b7d3-129f80e23881\") "
Oct 30 21:31:59.931049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:59.931017    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d909606f-340f-471b-b7d3-129f80e23881-maas-api-openshift-ca" pod "d909606f-340f-471b-b7d3-129f80e23881" (UID: "d909606f-340f-471b-b7d3-129f80e23881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:59.932465 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:59.932443    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d909606f-340f-471b-b7d3-129f80e23881-maas-api-xks-ca" pod "d909606f-340f-471b-b7d3-129f80e23881" (UID: "d909606f-340f-471b-b7d3-129f80e23881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:59.932751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:31:59.932736    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d909606f-340f-471b-b7d3-129f80e23881-kube-api-access-hq4vr" pod "d909606f-340f-471b-b7d3-129f80e23881" (UID: "d909606f-340f-471b-b7d3-129f80e23881"). InnerVolumeSpecName "kube-api-access-hq4vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:00.032240 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:00.032206    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq4vr\" (UniqueName: \"kubernetes.io/projected/d909606f-340f-471b-b7d3-129f80e23881-kube-api-access-hq4vr\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:00.032240 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:00.032236    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d909606f-340f-471b-b7d3-129f80e23881-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:00.032385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:00.032252    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d909606f-340f-471b-b7d3-129f80e23881-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:00.686966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:00.686932    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"
Oct 30 21:32:00.686966 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:00.686941    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44" event={"ID":"d909606f-340f-471b-b7d3-129f80e23881","Type":"ContainerDied","Data":"42fa3c5b6fba1e482f869b3e05cd25f04e6c51acc3bac1dd540b807e1f4a21ba"}
Oct 30 21:32:00.687504 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:00.687034    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42fa3c5b6fba1e482f869b3e05cd25f04e6c51acc3bac1dd540b807e1f4a21ba"
Oct 30 21:32:01.138179 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:01.138141    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd" podUID="9846a273-e5af-4bfc-b365-0d3fb804e5c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:32:02.547909 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:02.545368    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"]
Oct 30 21:32:02.549112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:02.549084    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c4052-hcfqf"]
Oct 30 21:32:03.661781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:03.661749    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e8e825-6cb3-4dd8-95ab-274ed6dbd62c" path="/var/lib/kubelet/pods/85e8e825-6cb3-4dd8-95ab-274ed6dbd62c/volumes"
Oct 30 21:32:05.285926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.285878    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc" podUID="60648dda-a037-461c-92d4-c9e110f1e312" containerName="main" containerID="cri-o://d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717" gracePeriod=45
Oct 30 21:32:05.616772 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.616742    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:32:05.740953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.732706    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60648dda-a037-461c-92d4-c9e110f1e312-tls-certs\" (UniqueName: \"kubernetes.io/secret/60648dda-a037-461c-92d4-c9e110f1e312-tls-certs\") pod \"60648dda-a037-461c-92d4-c9e110f1e312\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") "
Oct 30 21:32:05.740953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.732907    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-model-cache\") pod \"60648dda-a037-461c-92d4-c9e110f1e312\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") "
Oct 30 21:32:05.740953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.732989    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60648dda-a037-461c-92d4-c9e110f1e312-kube-api-access-vqcg8\" (UniqueName: \"kubernetes.io/projected/60648dda-a037-461c-92d4-c9e110f1e312-kube-api-access-vqcg8\") pod \"60648dda-a037-461c-92d4-c9e110f1e312\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") "
Oct 30 21:32:05.740953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.733171    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-tmp-dir\") pod \"60648dda-a037-461c-92d4-c9e110f1e312\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") "
Oct 30 21:32:05.740953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.733215    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-home\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-home\") pod \"60648dda-a037-461c-92d4-c9e110f1e312\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") "
Oct 30 21:32:05.740953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.733362    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-dshm\") pod \"60648dda-a037-461c-92d4-c9e110f1e312\" (UID: \"60648dda-a037-461c-92d4-c9e110f1e312\") "
Oct 30 21:32:05.740953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.737271    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-dshm" pod "60648dda-a037-461c-92d4-c9e110f1e312" (UID: "60648dda-a037-461c-92d4-c9e110f1e312"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:05.746890 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.746697    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:05.747265 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.747219    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-home" pod "60648dda-a037-461c-92d4-c9e110f1e312" (UID: "60648dda-a037-461c-92d4-c9e110f1e312"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:05.747737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.747708    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-model-cache" pod "60648dda-a037-461c-92d4-c9e110f1e312" (UID: "60648dda-a037-461c-92d4-c9e110f1e312"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:05.747958 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.747937    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-tmp-dir" pod "60648dda-a037-461c-92d4-c9e110f1e312" (UID: "60648dda-a037-461c-92d4-c9e110f1e312"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:05.749391 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.749326    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60648dda-a037-461c-92d4-c9e110f1e312-kube-api-access-vqcg8" pod "60648dda-a037-461c-92d4-c9e110f1e312" (UID: "60648dda-a037-461c-92d4-c9e110f1e312"). InnerVolumeSpecName "kube-api-access-vqcg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:05.760289 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.760257    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60648dda-a037-461c-92d4-c9e110f1e312-tls-certs" pod "60648dda-a037-461c-92d4-c9e110f1e312" (UID: "60648dda-a037-461c-92d4-c9e110f1e312"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:05.762553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.762520    4810 generic.go:356] "Generic (PLEG): container finished" podID="60648dda-a037-461c-92d4-c9e110f1e312" containerID="d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717" exitCode=0
Oct 30 21:32:05.762815 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.762799    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"
Oct 30 21:32:05.763800 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.762781    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc" event={"ID":"60648dda-a037-461c-92d4-c9e110f1e312","Type":"ContainerDied","Data":"d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717"}
Oct 30 21:32:05.763968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.763828    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc" event={"ID":"60648dda-a037-461c-92d4-c9e110f1e312","Type":"ContainerDied","Data":"5552c64c3bc86ac0dc682f19ec57a8c452b167f3fc6bade472a62582874c29fb"}
Oct 30 21:32:05.763968 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.763887    4810 scope.go:122] "RemoveContainer" containerID="d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717"
Oct 30 21:32:05.780302 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.780285    4810 scope.go:122] "RemoveContainer" containerID="d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717"
Oct 30 21:32:05.780647 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:05.780620    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717\": container with ID starting with d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717 not found: ID does not exist" containerID="d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717"
Oct 30 21:32:05.780683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.780660    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717"} err="failed to get container status \"d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717\": rpc error: code = NotFound desc = could not find container \"d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717\": container with ID starting with d05fc6b4fb50916a2b303e5f6b2c7d4b3f35b571390903a2327f2ca9aee7e717 not found: ID does not exist"
Oct 30 21:32:05.802136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.802105    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"]
Oct 30 21:32:05.827747 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.827680    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f8e2c3/test-model-f8e2c3-kserve-955d688d9-5tlvc"]
Oct 30 21:32:05.848226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.848195    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60648dda-a037-461c-92d4-c9e110f1e312-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:05.848226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.848221    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:05.848386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.848233    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqcg8\" (UniqueName: \"kubernetes.io/projected/60648dda-a037-461c-92d4-c9e110f1e312-kube-api-access-vqcg8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:05.848386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.848242    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:05.848386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:05.848252    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60648dda-a037-461c-92d4-c9e110f1e312-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:06.091247 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.091166    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"]
Oct 30 21:32:06.091839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.091818    4810 cpu_manager.go:414] "containerMap: removing container" podUID="d909606f-340f-471b-b7d3-129f80e23881" containerName="revoke-keys"
Oct 30 21:32:06.091972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.091840    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d909606f-340f-471b-b7d3-129f80e23881" containerName="revoke-keys"
Oct 30 21:32:06.091972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.091959    4810 cpu_manager.go:414] "containerMap: removing container" podUID="60648dda-a037-461c-92d4-c9e110f1e312" containerName="main"
Oct 30 21:32:06.091972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.091969    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60648dda-a037-461c-92d4-c9e110f1e312" containerName="main"
Oct 30 21:32:06.092160 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.092144    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="60648dda-a037-461c-92d4-c9e110f1e312" containerName="main"
Oct 30 21:32:06.092220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.092162    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="d909606f-340f-471b-b7d3-129f80e23881" containerName="revoke-keys"
Oct 30 21:32:06.096908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.096887    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.102982 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.102958    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"]
Oct 30 21:32:06.136330 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.136300    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:06.136534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.136514    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:32:06.152139 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.152103    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7h4c\" (UniqueName: \"kubernetes.io/projected/18906d17-4250-4cb6-be83-8d10f7ae6a24-kube-api-access-m7h4c\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.152323 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.152258    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.152323 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.152318    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.213286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.213249    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:32:06.267095 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.255281    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7h4c\" (UniqueName: \"kubernetes.io/projected/18906d17-4250-4cb6-be83-8d10f7ae6a24-kube-api-access-m7h4c\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.267095 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.255462    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.267095 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.255557    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.267095 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.256442    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.309710 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.309612    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7h4c\" (UniqueName: \"kubernetes.io/projected/18906d17-4250-4cb6-be83-8d10f7ae6a24-kube-api-access-m7h4c\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.314654 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.314624    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.418653 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.418610    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:06.706121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.706091    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"]
Oct 30 21:32:06.709441 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:06.709397    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18906d17_4250_4cb6_be83_8d10f7ae6a24.slice/crio-19c3816eee8604749c672f23222caae976b104e2fa9c45b3d152d70e6cf6a4cc WatchSource:0}: Error finding container 19c3816eee8604749c672f23222caae976b104e2fa9c45b3d152d70e6cf6a4cc: Status 404 returned error can't find the container with id 19c3816eee8604749c672f23222caae976b104e2fa9c45b3d152d70e6cf6a4cc
Oct 30 21:32:06.773817 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:06.773777    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p" event={"ID":"18906d17-4250-4cb6-be83-8d10f7ae6a24","Type":"ContainerStarted","Data":"19c3816eee8604749c672f23222caae976b104e2fa9c45b3d152d70e6cf6a4cc"}
Oct 30 21:32:07.662918 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:07.662883    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60648dda-a037-461c-92d4-c9e110f1e312" path="/var/lib/kubelet/pods/60648dda-a037-461c-92d4-c9e110f1e312/volumes"
Oct 30 21:32:07.785806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:07.785766    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p" event={"ID":"18906d17-4250-4cb6-be83-8d10f7ae6a24","Type":"ContainerStarted","Data":"e01d8fcef44182ae3c074f170b21dc694ad60acc6bdf1eafb8a8c05ed609d307"}
Oct 30 21:32:07.804338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:07.804292    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p" podStartSLOduration=1.804278245 podStartE2EDuration="1.804278245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:07.801223512 +0000 UTC m=+3294.293040057" watchObservedRunningTime="2026-10-30 21:32:07.804278245 +0000 UTC m=+3294.296094827"
Oct 30 21:32:08.443687 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.443650    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"]
Oct 30 21:32:08.447552 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.447535    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.458898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.458536    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"]
Oct 30 21:32:08.598501 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.598462    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.598714 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.598578    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.598714 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.598615    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64xcv\" (UniqueName: \"kubernetes.io/projected/7eef0c04-e32d-46f5-b691-5b7256b84214-kube-api-access-64xcv\") pod \"maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.699751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.699654    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.700257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.699798    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.700257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.699824    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64xcv\" (UniqueName: \"kubernetes.io/projected/7eef0c04-e32d-46f5-b691-5b7256b84214-kube-api-access-64xcv\") pod \"maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.700404 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.700374    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.702375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.702347    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.708275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.708247    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64xcv\" (UniqueName: \"kubernetes.io/projected/7eef0c04-e32d-46f5-b691-5b7256b84214-kube-api-access-64xcv\") pod \"maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.763985 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.763956    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:08.798131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.798097    4810 generic.go:356] "Generic (PLEG): container finished" podID="18906d17-4250-4cb6-be83-8d10f7ae6a24" containerID="e01d8fcef44182ae3c074f170b21dc694ad60acc6bdf1eafb8a8c05ed609d307" exitCode=0
Oct 30 21:32:08.798314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:08.798156    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p" event={"ID":"18906d17-4250-4cb6-be83-8d10f7ae6a24","Type":"ContainerDied","Data":"e01d8fcef44182ae3c074f170b21dc694ad60acc6bdf1eafb8a8c05ed609d307"}
Oct 30 21:32:09.057389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:09.057300    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"]
Oct 30 21:32:09.060486 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:09.060458    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eef0c04_e32d_46f5_b691_5b7256b84214.slice/crio-4ed14ca5a5897da2c41333e524f6e31a31d4a09254e778ac08e69ccc730089df WatchSource:0}: Error finding container 4ed14ca5a5897da2c41333e524f6e31a31d4a09254e778ac08e69ccc730089df: Status 404 returned error can't find the container with id 4ed14ca5a5897da2c41333e524f6e31a31d4a09254e778ac08e69ccc730089df
Oct 30 21:32:09.809050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:09.808950    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph" event={"ID":"7eef0c04-e32d-46f5-b691-5b7256b84214","Type":"ContainerStarted","Data":"ec40ca4cf8ed906a49fb51c3e1f4bfc137dc6d7db26eb87742d9faa96cdf088e"}
Oct 30 21:32:09.809050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:09.809015    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph" event={"ID":"7eef0c04-e32d-46f5-b691-5b7256b84214","Type":"ContainerStarted","Data":"4ed14ca5a5897da2c41333e524f6e31a31d4a09254e778ac08e69ccc730089df"}
Oct 30 21:32:09.830984 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:09.830926    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph" podStartSLOduration=1.830907513 podStartE2EDuration="1.830907513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:09.827783777 +0000 UTC m=+3296.319600323" watchObservedRunningTime="2026-10-30 21:32:09.830907513 +0000 UTC m=+3296.322724059"
Oct 30 21:32:09.988449 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:09.988143    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:10.118494 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.118409    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18906d17-4250-4cb6-be83-8d10f7ae6a24-kube-api-access-m7h4c\" (UniqueName: \"kubernetes.io/projected/18906d17-4250-4cb6-be83-8d10f7ae6a24-kube-api-access-m7h4c\") pod \"18906d17-4250-4cb6-be83-8d10f7ae6a24\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") "
Oct 30 21:32:10.118679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.118489    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-xks-ca\") pod \"18906d17-4250-4cb6-be83-8d10f7ae6a24\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") "
Oct 30 21:32:10.118679 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.118559    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-openshift-ca\") pod \"18906d17-4250-4cb6-be83-8d10f7ae6a24\" (UID: \"18906d17-4250-4cb6-be83-8d10f7ae6a24\") "
Oct 30 21:32:10.119569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.119539    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-openshift-ca" pod "18906d17-4250-4cb6-be83-8d10f7ae6a24" (UID: "18906d17-4250-4cb6-be83-8d10f7ae6a24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:10.121756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.121721    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-xks-ca" pod "18906d17-4250-4cb6-be83-8d10f7ae6a24" (UID: "18906d17-4250-4cb6-be83-8d10f7ae6a24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:10.121896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.121809    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18906d17-4250-4cb6-be83-8d10f7ae6a24-kube-api-access-m7h4c" pod "18906d17-4250-4cb6-be83-8d10f7ae6a24" (UID: "18906d17-4250-4cb6-be83-8d10f7ae6a24"). InnerVolumeSpecName "kube-api-access-m7h4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:10.220649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.220610    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:10.220649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.220644    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7h4c\" (UniqueName: \"kubernetes.io/projected/18906d17-4250-4cb6-be83-8d10f7ae6a24-kube-api-access-m7h4c\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:10.220902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.220660    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18906d17-4250-4cb6-be83-8d10f7ae6a24-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:10.739144 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.739115    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"]
Oct 30 21:32:10.739725 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.739712    4810 cpu_manager.go:414] "containerMap: removing container" podUID="18906d17-4250-4cb6-be83-8d10f7ae6a24" containerName="revoke-keys"
Oct 30 21:32:10.739781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.739726    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18906d17-4250-4cb6-be83-8d10f7ae6a24" containerName="revoke-keys"
Oct 30 21:32:10.739845 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.739835    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="18906d17-4250-4cb6-be83-8d10f7ae6a24" containerName="revoke-keys"
Oct 30 21:32:10.743911 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.743891    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:10.751164 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.751136    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"]
Oct 30 21:32:10.822171 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.822143    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"
Oct 30 21:32:10.822649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.822177    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p" event={"ID":"18906d17-4250-4cb6-be83-8d10f7ae6a24","Type":"ContainerDied","Data":"19c3816eee8604749c672f23222caae976b104e2fa9c45b3d152d70e6cf6a4cc"}
Oct 30 21:32:10.822649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.822226    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c3816eee8604749c672f23222caae976b104e2fa9c45b3d152d70e6cf6a4cc"
Oct 30 21:32:10.824010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.823985    4810 generic.go:356] "Generic (PLEG): container finished" podID="7eef0c04-e32d-46f5-b691-5b7256b84214" containerID="ec40ca4cf8ed906a49fb51c3e1f4bfc137dc6d7db26eb87742d9faa96cdf088e" exitCode=0
Oct 30 21:32:10.824146 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.824067    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph" event={"ID":"7eef0c04-e32d-46f5-b691-5b7256b84214","Type":"ContainerDied","Data":"ec40ca4cf8ed906a49fb51c3e1f4bfc137dc6d7db26eb87742d9faa96cdf088e"}
Oct 30 21:32:10.829407 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.829383    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:10.829505 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.829449    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:10.829606 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.829584    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkb8\" (UniqueName: \"kubernetes.io/projected/a9491b77-14b7-4a13-ac6e-da35c11ea18d-kube-api-access-qwkb8\") pod \"maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:10.930825 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.930781    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:10.931031 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.930926    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkb8\" (UniqueName: \"kubernetes.io/projected/a9491b77-14b7-4a13-ac6e-da35c11ea18d-kube-api-access-qwkb8\") pod \"maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:10.931098 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.931066    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:10.932269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.932237    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:10.934065 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.934037    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:10.949034 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:10.949009    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkb8\" (UniqueName: \"kubernetes.io/projected/a9491b77-14b7-4a13-ac6e-da35c11ea18d-kube-api-access-qwkb8\") pod \"maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:11.061070 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:11.061000    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:11.138669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:11.138630    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd" podUID="9846a273-e5af-4bfc-b365-0d3fb804e5c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:32:11.349400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:11.349370    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"]
Oct 30 21:32:11.351141 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:11.351112    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9491b77_14b7_4a13_ac6e_da35c11ea18d.slice/crio-155ff242d1cc4f87bc7a815c61d2ce6b1793ba1adc9b69a6da4e442cf6c08eee WatchSource:0}: Error finding container 155ff242d1cc4f87bc7a815c61d2ce6b1793ba1adc9b69a6da4e442cf6c08eee: Status 404 returned error can't find the container with id 155ff242d1cc4f87bc7a815c61d2ce6b1793ba1adc9b69a6da4e442cf6c08eee
Oct 30 21:32:11.833516 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:11.833464    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv" event={"ID":"a9491b77-14b7-4a13-ac6e-da35c11ea18d","Type":"ContainerStarted","Data":"ac996c035ab90a402e6f3952fae0a22d746d4f0bb83e56307d2c96fbb45ecaee"}
Oct 30 21:32:11.834049 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:11.833532    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv" event={"ID":"a9491b77-14b7-4a13-ac6e-da35c11ea18d","Type":"ContainerStarted","Data":"155ff242d1cc4f87bc7a815c61d2ce6b1793ba1adc9b69a6da4e442cf6c08eee"}
Oct 30 21:32:11.853644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:11.853587    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv" podStartSLOduration=1.8535742640000001 podStartE2EDuration="1.853574264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:11.849847289 +0000 UTC m=+3298.341663836" watchObservedRunningTime="2026-10-30 21:32:11.853574264 +0000 UTC m=+3298.345390808"
Oct 30 21:32:12.022669 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.022640    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:12.147427 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.147394    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-openshift-ca\") pod \"7eef0c04-e32d-46f5-b691-5b7256b84214\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") "
Oct 30 21:32:12.147579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.147518    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-xks-ca\") pod \"7eef0c04-e32d-46f5-b691-5b7256b84214\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") "
Oct 30 21:32:12.147731 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.147711    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eef0c04-e32d-46f5-b691-5b7256b84214-kube-api-access-64xcv\" (UniqueName: \"kubernetes.io/projected/7eef0c04-e32d-46f5-b691-5b7256b84214-kube-api-access-64xcv\") pod \"7eef0c04-e32d-46f5-b691-5b7256b84214\" (UID: \"7eef0c04-e32d-46f5-b691-5b7256b84214\") "
Oct 30 21:32:12.147840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.147818    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-openshift-ca" pod "7eef0c04-e32d-46f5-b691-5b7256b84214" (UID: "7eef0c04-e32d-46f5-b691-5b7256b84214"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:12.148131 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.148113    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:12.149644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.149621    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-xks-ca" pod "7eef0c04-e32d-46f5-b691-5b7256b84214" (UID: "7eef0c04-e32d-46f5-b691-5b7256b84214"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:12.149888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.149853    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eef0c04-e32d-46f5-b691-5b7256b84214-kube-api-access-64xcv" pod "7eef0c04-e32d-46f5-b691-5b7256b84214" (UID: "7eef0c04-e32d-46f5-b691-5b7256b84214"). InnerVolumeSpecName "kube-api-access-64xcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:12.249416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.249376    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eef0c04-e32d-46f5-b691-5b7256b84214-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:12.249416 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.249408    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64xcv\" (UniqueName: \"kubernetes.io/projected/7eef0c04-e32d-46f5-b691-5b7256b84214-kube-api-access-64xcv\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:12.842083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.842050    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"
Oct 30 21:32:12.842530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.842050    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph" event={"ID":"7eef0c04-e32d-46f5-b691-5b7256b84214","Type":"ContainerDied","Data":"4ed14ca5a5897da2c41333e524f6e31a31d4a09254e778ac08e69ccc730089df"}
Oct 30 21:32:12.842530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.842215    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ed14ca5a5897da2c41333e524f6e31a31d4a09254e778ac08e69ccc730089df"
Oct 30 21:32:12.843734 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.843706    4810 generic.go:356] "Generic (PLEG): container finished" podID="a9491b77-14b7-4a13-ac6e-da35c11ea18d" containerID="ac996c035ab90a402e6f3952fae0a22d746d4f0bb83e56307d2c96fbb45ecaee" exitCode=0
Oct 30 21:32:12.843875 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:12.843733    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv" event={"ID":"a9491b77-14b7-4a13-ac6e-da35c11ea18d","Type":"ContainerDied","Data":"ac996c035ab90a402e6f3952fae0a22d746d4f0bb83e56307d2c96fbb45ecaee"}
Oct 30 21:32:13.179096 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:13.179059    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"]
Oct 30 21:32:14.026272 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.026235    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:14.112539 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.112503    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:32:14.175944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.175899    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9491b77-14b7-4a13-ac6e-da35c11ea18d-kube-api-access-qwkb8\" (UniqueName: \"kubernetes.io/projected/a9491b77-14b7-4a13-ac6e-da35c11ea18d-kube-api-access-qwkb8\") pod \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") "
Oct 30 21:32:14.176150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.175986    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-openshift-ca\") pod \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") "
Oct 30 21:32:14.176150 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.176054    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-xks-ca\") pod \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\" (UID: \"a9491b77-14b7-4a13-ac6e-da35c11ea18d\") "
Oct 30 21:32:14.177059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.177035    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-openshift-ca" pod "a9491b77-14b7-4a13-ac6e-da35c11ea18d" (UID: "a9491b77-14b7-4a13-ac6e-da35c11ea18d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:14.178672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.178641    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9491b77-14b7-4a13-ac6e-da35c11ea18d-kube-api-access-qwkb8" pod "a9491b77-14b7-4a13-ac6e-da35c11ea18d" (UID: "a9491b77-14b7-4a13-ac6e-da35c11ea18d"). InnerVolumeSpecName "kube-api-access-qwkb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:14.179155 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.179133    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-xks-ca" pod "a9491b77-14b7-4a13-ac6e-da35c11ea18d" (UID: "a9491b77-14b7-4a13-ac6e-da35c11ea18d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:14.277520 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.277436    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9846a273-e5af-4bfc-b365-0d3fb804e5c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9846a273-e5af-4bfc-b365-0d3fb804e5c4-plugins-config-volume\") pod \"9846a273-e5af-4bfc-b365-0d3fb804e5c4\" (UID: \"9846a273-e5af-4bfc-b365-0d3fb804e5c4\") "
Oct 30 21:32:14.277708 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.277519    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9846a273-e5af-4bfc-b365-0d3fb804e5c4-kube-api-access-hgkcd\" (UniqueName: \"kubernetes.io/projected/9846a273-e5af-4bfc-b365-0d3fb804e5c4-kube-api-access-hgkcd\") pod \"9846a273-e5af-4bfc-b365-0d3fb804e5c4\" (UID: \"9846a273-e5af-4bfc-b365-0d3fb804e5c4\") "
Oct 30 21:32:14.277920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.277904    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:14.277999 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.277923    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwkb8\" (UniqueName: \"kubernetes.io/projected/a9491b77-14b7-4a13-ac6e-da35c11ea18d-kube-api-access-qwkb8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:14.277999 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.277915    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9846a273-e5af-4bfc-b365-0d3fb804e5c4-plugins-config-volume" pod "9846a273-e5af-4bfc-b365-0d3fb804e5c4" (UID: "9846a273-e5af-4bfc-b365-0d3fb804e5c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:14.277999 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.277933    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9491b77-14b7-4a13-ac6e-da35c11ea18d-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:14.279748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.279722    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9846a273-e5af-4bfc-b365-0d3fb804e5c4-kube-api-access-hgkcd" pod "9846a273-e5af-4bfc-b365-0d3fb804e5c4" (UID: "9846a273-e5af-4bfc-b365-0d3fb804e5c4"). InnerVolumeSpecName "kube-api-access-hgkcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:14.379526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.379494    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgkcd\" (UniqueName: \"kubernetes.io/projected/9846a273-e5af-4bfc-b365-0d3fb804e5c4-kube-api-access-hgkcd\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:14.379526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.379527    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9846a273-e5af-4bfc-b365-0d3fb804e5c4-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:14.863357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.863324    4810 generic.go:356] "Generic (PLEG): container finished" podID="9846a273-e5af-4bfc-b365-0d3fb804e5c4" containerID="8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe" exitCode=0
Oct 30 21:32:14.863549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.863373    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"
Oct 30 21:32:14.863549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.863420    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd" event={"ID":"9846a273-e5af-4bfc-b365-0d3fb804e5c4","Type":"ContainerDied","Data":"8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe"}
Oct 30 21:32:14.863549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.863487    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd" event={"ID":"9846a273-e5af-4bfc-b365-0d3fb804e5c4","Type":"ContainerDied","Data":"adb712f54a1c17487b4e6410daa934019fe9daa9e0b5a81bb6192f5ac611d935"}
Oct 30 21:32:14.863549 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.863520    4810 scope.go:122] "RemoveContainer" containerID="8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe"
Oct 30 21:32:14.865019 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.865004    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"
Oct 30 21:32:14.865108 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.865052    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv" event={"ID":"a9491b77-14b7-4a13-ac6e-da35c11ea18d","Type":"ContainerDied","Data":"155ff242d1cc4f87bc7a815c61d2ce6b1793ba1adc9b69a6da4e442cf6c08eee"}
Oct 30 21:32:14.865108 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.865101    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="155ff242d1cc4f87bc7a815c61d2ce6b1793ba1adc9b69a6da4e442cf6c08eee"
Oct 30 21:32:14.878005 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.877986    4810 scope.go:122] "RemoveContainer" containerID="8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe"
Oct 30 21:32:14.878320 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:14.878296    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe\": container with ID starting with 8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe not found: ID does not exist" containerID="8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe"
Oct 30 21:32:14.878422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.878326    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe"} err="failed to get container status \"8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe\": rpc error: code = NotFound desc = could not find container \"8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe\": container with ID starting with 8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe not found: ID does not exist"
Oct 30 21:32:14.898557 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.898531    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"]
Oct 30 21:32:14.905087 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.905058    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8fa15b-c5f767997-jxnrd"]
Oct 30 21:32:14.930422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.930397    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:32:14.932777 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:14.932757    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:32:15.662878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:15.662819    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9846a273-e5af-4bfc-b365-0d3fb804e5c4" path="/var/lib/kubelet/pods/9846a273-e5af-4bfc-b365-0d3fb804e5c4/volumes"
Oct 30 21:32:16.217590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:16.217525    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"]
Oct 30 21:32:16.223442 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:16.223403    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a50675-p5t44"]
Oct 30 21:32:16.334458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:16.334423    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"]
Oct 30 21:32:16.350200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:16.350167    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"]
Oct 30 21:32:16.748898 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:16.748827    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:32:16.749369 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:16.749054    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:32:17.139715 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:17.139675    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:32:17.662444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:17.662410    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d909606f-340f-471b-b7d3-129f80e23881" path="/var/lib/kubelet/pods/d909606f-340f-471b-b7d3-129f80e23881/volumes"
Oct 30 21:32:19.571685 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:19.571653    4810 scope.go:122] "RemoveContainer" containerID="631e9a87184784f9231f9f598bef46ffee5a6f29f22478ed9d90081117c9c611"
Oct 30 21:32:19.582649 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:19.582622    4810 scope.go:122] "RemoveContainer" containerID="084ae85317acf352a652d4fdf6efe0e796a9f41b42e0bbb054a7d8b0b04d0951"
Oct 30 21:32:19.594431 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:19.594406    4810 scope.go:122] "RemoveContainer" containerID="dff56640367a5f408bdf63da5fd2e043717076f03c735cfd5a19a98316b8f9bc"
Oct 30 21:32:19.604486 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:19.604463    4810 scope.go:122] "RemoveContainer" containerID="9998550cf3be3ef6344bdcaf6c6d586c1c9edc2fc860351a3b107d30af2bb10d"
Oct 30 21:32:21.335193 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:21.335152    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb" podUID="0e472de9-60b5-47ae-9889-8540174d0bbf" containerName="payload-processing" containerID="cri-o://6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27" gracePeriod=25
Oct 30 21:32:21.350757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:21.350710    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b" podUID="9ac2c41a-2d51-4b93-acd2-55e9271b45ab" containerName="payload-pre-processing" containerID="cri-o://f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c" gracePeriod=25
Oct 30 21:32:22.434838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:22.434676    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"]
Oct 30 21:32:22.437566 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:22.437545    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f64640-zh69p"]
Oct 30 21:32:22.829587 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:22.829502    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:32:23.661707 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:23.661674    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18906d17-4250-4cb6-be83-8d10f7ae6a24" path="/var/lib/kubelet/pods/18906d17-4250-4cb6-be83-8d10f7ae6a24/volumes"
Oct 30 21:32:23.792357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:23.792313    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:32:24.000062 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:23.999972    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"]
Oct 30 21:32:24.000592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.000577    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a9491b77-14b7-4a13-ac6e-da35c11ea18d" containerName="revoke-keys"
Oct 30 21:32:24.000645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.000593    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9491b77-14b7-4a13-ac6e-da35c11ea18d" containerName="revoke-keys"
Oct 30 21:32:24.000645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.000607    4810 cpu_manager.go:414] "containerMap: removing container" podUID="7eef0c04-e32d-46f5-b691-5b7256b84214" containerName="revoke-keys"
Oct 30 21:32:24.000645 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.000613    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eef0c04-e32d-46f5-b691-5b7256b84214" containerName="revoke-keys"
Oct 30 21:32:24.000750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.000645    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9846a273-e5af-4bfc-b365-0d3fb804e5c4" containerName="payload-processing"
Oct 30 21:32:24.000750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.000653    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9846a273-e5af-4bfc-b365-0d3fb804e5c4" containerName="payload-processing"
Oct 30 21:32:24.000750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.000749    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9491b77-14b7-4a13-ac6e-da35c11ea18d" containerName="revoke-keys"
Oct 30 21:32:24.000882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.000761    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eef0c04-e32d-46f5-b691-5b7256b84214" containerName="revoke-keys"
Oct 30 21:32:24.000882 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.000770    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9846a273-e5af-4bfc-b365-0d3fb804e5c4" containerName="payload-processing"
Oct 30 21:32:24.007235 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.007212    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.011417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.011394    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"]
Oct 30 21:32:24.016828 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.016308    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.017995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.017972    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"]
Oct 30 21:32:24.022898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.022873    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"]
Oct 30 21:32:24.096363 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.096329    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:24.096561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.096391    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:32:24.111298 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.111268    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41b309d-c7jkz\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.111430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.111310    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8k2\" (UniqueName: \"kubernetes.io/projected/90687157-a544-4113-aae2-b4c0278794d4-kube-api-access-cm8k2\") pod \"maas-api-revoke-sub-cd30de49-k74nc\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.111430 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.111349    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90687157-a544-4113-aae2-b4c0278794d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd30de49-k74nc\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.111520 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.111490    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41b309d-c7jkz\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.111562 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.111527    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90687157-a544-4113-aae2-b4c0278794d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd30de49-k74nc\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.111619 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.111601    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmftb\" (UniqueName: \"kubernetes.io/projected/49f45b0d-c112-4592-9d24-1f1002eea388-kube-api-access-bmftb\") pod \"maas-api-revoke-sub-d41b309d-c7jkz\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.212838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.212805    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90687157-a544-4113-aae2-b4c0278794d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd30de49-k74nc\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.213037 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.212888    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41b309d-c7jkz\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.213037 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.212917    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90687157-a544-4113-aae2-b4c0278794d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd30de49-k74nc\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.213037 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.212938    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmftb\" (UniqueName: \"kubernetes.io/projected/49f45b0d-c112-4592-9d24-1f1002eea388-kube-api-access-bmftb\") pod \"maas-api-revoke-sub-d41b309d-c7jkz\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.213229 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.213109    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41b309d-c7jkz\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.213229 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.213156    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8k2\" (UniqueName: \"kubernetes.io/projected/90687157-a544-4113-aae2-b4c0278794d4-kube-api-access-cm8k2\") pod \"maas-api-revoke-sub-cd30de49-k74nc\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.213598 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.213572    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90687157-a544-4113-aae2-b4c0278794d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd30de49-k74nc\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.213840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.213821    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41b309d-c7jkz\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.215463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.215440    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41b309d-c7jkz\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.215553 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.215526    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90687157-a544-4113-aae2-b4c0278794d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd30de49-k74nc\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.224836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.224812    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8k2\" (UniqueName: \"kubernetes.io/projected/90687157-a544-4113-aae2-b4c0278794d4-kube-api-access-cm8k2\") pod \"maas-api-revoke-sub-cd30de49-k74nc\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.224994 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.224975    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmftb\" (UniqueName: \"kubernetes.io/projected/49f45b0d-c112-4592-9d24-1f1002eea388-kube-api-access-bmftb\") pod \"maas-api-revoke-sub-d41b309d-c7jkz\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.325535 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.325459    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:24.338789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.338763    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:24.640410 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.640376    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"]
Oct 30 21:32:24.644492 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:24.644465    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f45b0d_c112_4592_9d24_1f1002eea388.slice/crio-553149cc604a1be83e17905f3cbe68c75b3304e8da9d4e4ce93e413e91f89ffc WatchSource:0}: Error finding container 553149cc604a1be83e17905f3cbe68c75b3304e8da9d4e4ce93e413e91f89ffc: Status 404 returned error can't find the container with id 553149cc604a1be83e17905f3cbe68c75b3304e8da9d4e4ce93e413e91f89ffc
Oct 30 21:32:24.655509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.655485    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"]
Oct 30 21:32:24.657003 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:24.656976    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90687157_a544_4113_aae2_b4c0278794d4.slice/crio-de64f8bd00d20e25642d82cf3592c90c411a68dc40c73e73c7e41e54487c00bf WatchSource:0}: Error finding container de64f8bd00d20e25642d82cf3592c90c411a68dc40c73e73c7e41e54487c00bf: Status 404 returned error can't find the container with id de64f8bd00d20e25642d82cf3592c90c411a68dc40c73e73c7e41e54487c00bf
Oct 30 21:32:24.800881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.799566    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"]
Oct 30 21:32:24.800881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.799818    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy" containerID="cri-o://344dff5d15669b63be50c58595f1c0ef947a8330051e6b7a2c291ca735e9caf4" gracePeriod=30
Oct 30 21:32:24.969759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.969668    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz" event={"ID":"49f45b0d-c112-4592-9d24-1f1002eea388","Type":"ContainerStarted","Data":"553149cc604a1be83e17905f3cbe68c75b3304e8da9d4e4ce93e413e91f89ffc"}
Oct 30 21:32:24.971209 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:24.971177    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc" event={"ID":"90687157-a544-4113-aae2-b4c0278794d4","Type":"ContainerStarted","Data":"de64f8bd00d20e25642d82cf3592c90c411a68dc40c73e73c7e41e54487c00bf"}
Oct 30 21:32:25.159980 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:25.159913    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:32:25.536426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.536387    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"]
Oct 30 21:32:25.541517 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.541491    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8e2c3-l79mv"]
Oct 30 21:32:25.666668 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.666624    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9491b77-14b7-4a13-ac6e-da35c11ea18d" path="/var/lib/kubelet/pods/a9491b77-14b7-4a13-ac6e-da35c11ea18d/volumes"
Oct 30 21:32:25.674490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.674459    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"]
Oct 30 21:32:25.679475 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.679447    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:25.684422 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.683933    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"]
Oct 30 21:32:25.737314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.737272    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:25.737521 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.737332    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:25.737521 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.737494    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxs4\" (UniqueName: \"kubernetes.io/projected/8906e24a-9968-4bcc-895e-6da393ac5ef4-kube-api-access-6zxs4\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:25.838680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.838650    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:25.839116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.838708    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:25.839116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.838759    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxs4\" (UniqueName: \"kubernetes.io/projected/8906e24a-9968-4bcc-895e-6da393ac5ef4-kube-api-access-6zxs4\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:25.839942 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.839816    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:25.841672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.841650    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:25.847597 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.847578    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxs4\" (UniqueName: \"kubernetes.io/projected/8906e24a-9968-4bcc-895e-6da393ac5ef4-kube-api-access-6zxs4\") pod \"maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:25.981694 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.981655    4810 generic.go:356] "Generic (PLEG): container finished" podID="90687157-a544-4113-aae2-b4c0278794d4" containerID="9cfb817107cc12ddeb3a6f05eb2f1709766dd37bc6bb278c1e8117a99a0ae3f2" exitCode=0
Oct 30 21:32:25.981951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.981733    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc" event={"ID":"90687157-a544-4113-aae2-b4c0278794d4","Type":"ContainerDied","Data":"9cfb817107cc12ddeb3a6f05eb2f1709766dd37bc6bb278c1e8117a99a0ae3f2"}
Oct 30 21:32:25.983662 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.983636    4810 generic.go:356] "Generic (PLEG): container finished" podID="49f45b0d-c112-4592-9d24-1f1002eea388" containerID="47d74a9e9f47fe6fd2f7242ca895be9dc54095fbbd7bb167f237cb76879ab2b8" exitCode=0
Oct 30 21:32:25.983800 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:25.983672    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz" event={"ID":"49f45b0d-c112-4592-9d24-1f1002eea388","Type":"ContainerDied","Data":"47d74a9e9f47fe6fd2f7242ca895be9dc54095fbbd7bb167f237cb76879ab2b8"}
Oct 30 21:32:26.001332 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:26.001296    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:26.287908 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:26.282837    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"]
Oct 30 21:32:26.287908 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:26.284282    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8906e24a_9968_4bcc_895e_6da393ac5ef4.slice/crio-6b6317b6030f58549182b4df543c061653fd05cb50992b09005f8defd89b431f WatchSource:0}: Error finding container 6b6317b6030f58549182b4df543c061653fd05cb50992b09005f8defd89b431f: Status 404 returned error can't find the container with id 6b6317b6030f58549182b4df543c061653fd05cb50992b09005f8defd89b431f
Oct 30 21:32:26.995845 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:26.995804    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z" event={"ID":"8906e24a-9968-4bcc-895e-6da393ac5ef4","Type":"ContainerStarted","Data":"339504c2458768677b1cd9ec0bfef67e16d670c4f78f445fbc823a7c1efa4e3c"}
Oct 30 21:32:26.996342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:26.995907    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z" event={"ID":"8906e24a-9968-4bcc-895e-6da393ac5ef4","Type":"ContainerStarted","Data":"6b6317b6030f58549182b4df543c061653fd05cb50992b09005f8defd89b431f"}
Oct 30 21:32:27.017510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.017433    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z" podStartSLOduration=2.017409795 podStartE2EDuration="2.017409795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:27.013747862 +0000 UTC m=+3313.505564407" watchObservedRunningTime="2026-10-30 21:32:27.017409795 +0000 UTC m=+3313.509226343"
Oct 30 21:32:27.163294 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.163258    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:27.239587 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.239556    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:27.255153 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.255038    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-xks-ca\") pod \"49f45b0d-c112-4592-9d24-1f1002eea388\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") "
Oct 30 21:32:27.255153 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.255134    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-openshift-ca\") pod \"49f45b0d-c112-4592-9d24-1f1002eea388\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") "
Oct 30 21:32:27.255383 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.255307    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f45b0d-c112-4592-9d24-1f1002eea388-kube-api-access-bmftb\" (UniqueName: \"kubernetes.io/projected/49f45b0d-c112-4592-9d24-1f1002eea388-kube-api-access-bmftb\") pod \"49f45b0d-c112-4592-9d24-1f1002eea388\" (UID: \"49f45b0d-c112-4592-9d24-1f1002eea388\") "
Oct 30 21:32:27.256021 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.255990    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-openshift-ca" pod "49f45b0d-c112-4592-9d24-1f1002eea388" (UID: "49f45b0d-c112-4592-9d24-1f1002eea388"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:27.257661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.257626    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-xks-ca" pod "49f45b0d-c112-4592-9d24-1f1002eea388" (UID: "49f45b0d-c112-4592-9d24-1f1002eea388"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:27.258527 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.258476    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f45b0d-c112-4592-9d24-1f1002eea388-kube-api-access-bmftb" pod "49f45b0d-c112-4592-9d24-1f1002eea388" (UID: "49f45b0d-c112-4592-9d24-1f1002eea388"). InnerVolumeSpecName "kube-api-access-bmftb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:27.356992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.356960    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90687157-a544-4113-aae2-b4c0278794d4-kube-api-access-cm8k2\" (UniqueName: \"kubernetes.io/projected/90687157-a544-4113-aae2-b4c0278794d4-kube-api-access-cm8k2\") pod \"90687157-a544-4113-aae2-b4c0278794d4\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") "
Oct 30 21:32:27.357205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.357011    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90687157-a544-4113-aae2-b4c0278794d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90687157-a544-4113-aae2-b4c0278794d4-maas-api-xks-ca\") pod \"90687157-a544-4113-aae2-b4c0278794d4\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") "
Oct 30 21:32:27.357205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.357076    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90687157-a544-4113-aae2-b4c0278794d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90687157-a544-4113-aae2-b4c0278794d4-maas-api-openshift-ca\") pod \"90687157-a544-4113-aae2-b4c0278794d4\" (UID: \"90687157-a544-4113-aae2-b4c0278794d4\") "
Oct 30 21:32:27.357488 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.357463    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90687157-a544-4113-aae2-b4c0278794d4-maas-api-openshift-ca" pod "90687157-a544-4113-aae2-b4c0278794d4" (UID: "90687157-a544-4113-aae2-b4c0278794d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:27.357587 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.357571    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:27.357652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.357587    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90687157-a544-4113-aae2-b4c0278794d4-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:27.357652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.357598    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f45b0d-c112-4592-9d24-1f1002eea388-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:27.357652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.357612    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmftb\" (UniqueName: \"kubernetes.io/projected/49f45b0d-c112-4592-9d24-1f1002eea388-kube-api-access-bmftb\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:27.359168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.359144    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90687157-a544-4113-aae2-b4c0278794d4-maas-api-xks-ca" pod "90687157-a544-4113-aae2-b4c0278794d4" (UID: "90687157-a544-4113-aae2-b4c0278794d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:27.359245 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.359190    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90687157-a544-4113-aae2-b4c0278794d4-kube-api-access-cm8k2" pod "90687157-a544-4113-aae2-b4c0278794d4" (UID: "90687157-a544-4113-aae2-b4c0278794d4"). InnerVolumeSpecName "kube-api-access-cm8k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:27.458682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.458642    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm8k2\" (UniqueName: \"kubernetes.io/projected/90687157-a544-4113-aae2-b4c0278794d4-kube-api-access-cm8k2\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:27.458682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:27.458677    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90687157-a544-4113-aae2-b4c0278794d4-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:28.008546 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.008447    4810 generic.go:356] "Generic (PLEG): container finished" podID="8906e24a-9968-4bcc-895e-6da393ac5ef4" containerID="339504c2458768677b1cd9ec0bfef67e16d670c4f78f445fbc823a7c1efa4e3c" exitCode=0
Oct 30 21:32:28.009020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.008531    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z" event={"ID":"8906e24a-9968-4bcc-895e-6da393ac5ef4","Type":"ContainerDied","Data":"339504c2458768677b1cd9ec0bfef67e16d670c4f78f445fbc823a7c1efa4e3c"}
Oct 30 21:32:28.010107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.010085    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz"
Oct 30 21:32:28.010261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.010084    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b309d-c7jkz" event={"ID":"49f45b0d-c112-4592-9d24-1f1002eea388","Type":"ContainerDied","Data":"553149cc604a1be83e17905f3cbe68c75b3304e8da9d4e4ce93e413e91f89ffc"}
Oct 30 21:32:28.010261 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.010208    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="553149cc604a1be83e17905f3cbe68c75b3304e8da9d4e4ce93e413e91f89ffc"
Oct 30 21:32:28.012979 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.012951    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc" event={"ID":"90687157-a544-4113-aae2-b4c0278794d4","Type":"ContainerDied","Data":"de64f8bd00d20e25642d82cf3592c90c411a68dc40c73e73c7e41e54487c00bf"}
Oct 30 21:32:28.013112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.012999    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de64f8bd00d20e25642d82cf3592c90c411a68dc40c73e73c7e41e54487c00bf"
Oct 30 21:32:28.013112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.013053    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd30de49-k74nc"
Oct 30 21:32:28.207050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.207014    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" containerID="cri-o://d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac" gracePeriod=45
Oct 30 21:32:28.299421 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.298698    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": dial tcp 10.132.0.59:8000: connect: connection refused"
Oct 30 21:32:28.483444 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.483411    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 21:32:28.575836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.575747    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-home\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-home\") pod \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") "
Oct 30 21:32:28.576058 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.575845    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tls-certs\" (UniqueName: \"kubernetes.io/secret/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tls-certs\") pod \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") "
Oct 30 21:32:28.576058 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.575920    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-model-cache\") pod \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") "
Oct 30 21:32:28.576058 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.575990    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tmp-dir\") pod \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") "
Oct 30 21:32:28.576226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.576141    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-dshm\") pod \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") "
Oct 30 21:32:28.576226 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.576193    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c1e45e-3f48-47e8-b216-77d2a7d6afca-kube-api-access-rdv9m\" (UniqueName: \"kubernetes.io/projected/71c1e45e-3f48-47e8-b216-77d2a7d6afca-kube-api-access-rdv9m\") pod \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\" (UID: \"71c1e45e-3f48-47e8-b216-77d2a7d6afca\") "
Oct 30 21:32:28.577163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.577034    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-model-cache" pod "71c1e45e-3f48-47e8-b216-77d2a7d6afca" (UID: "71c1e45e-3f48-47e8-b216-77d2a7d6afca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:28.577438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.577408    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tmp-dir" pod "71c1e45e-3f48-47e8-b216-77d2a7d6afca" (UID: "71c1e45e-3f48-47e8-b216-77d2a7d6afca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:28.577667 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.577641    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-home" pod "71c1e45e-3f48-47e8-b216-77d2a7d6afca" (UID: "71c1e45e-3f48-47e8-b216-77d2a7d6afca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:28.580849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.580802    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tls-certs" pod "71c1e45e-3f48-47e8-b216-77d2a7d6afca" (UID: "71c1e45e-3f48-47e8-b216-77d2a7d6afca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:28.580849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.580812    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-dshm" pod "71c1e45e-3f48-47e8-b216-77d2a7d6afca" (UID: "71c1e45e-3f48-47e8-b216-77d2a7d6afca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:28.589007 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.588760    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c1e45e-3f48-47e8-b216-77d2a7d6afca-kube-api-access-rdv9m" pod "71c1e45e-3f48-47e8-b216-77d2a7d6afca" (UID: "71c1e45e-3f48-47e8-b216-77d2a7d6afca"). InnerVolumeSpecName "kube-api-access-rdv9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:28.677750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.677712    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:28.677750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.677747    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:28.677750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.677757    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdv9m\" (UniqueName: \"kubernetes.io/projected/71c1e45e-3f48-47e8-b216-77d2a7d6afca-kube-api-access-rdv9m\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:28.678084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.677768    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:28.678084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.677778    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71c1e45e-3f48-47e8-b216-77d2a7d6afca-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:28.678084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:28.677786    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71c1e45e-3f48-47e8-b216-77d2a7d6afca-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:29.021158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.021118    4810 generic.go:356] "Generic (PLEG): container finished" podID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerID="d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac" exitCode=0
Oct 30 21:32:29.021158 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.021157    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"
Oct 30 21:32:29.021763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.021187    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" event={"ID":"71c1e45e-3f48-47e8-b216-77d2a7d6afca","Type":"ContainerDied","Data":"d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac"}
Oct 30 21:32:29.021763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.021232    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w" event={"ID":"71c1e45e-3f48-47e8-b216-77d2a7d6afca","Type":"ContainerDied","Data":"d48389cf48b9f6889de73b94ab33714c6849fb4fa4fa355b12503a96c32766ba"}
Oct 30 21:32:29.021763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.021249    4810 scope.go:122] "RemoveContainer" containerID="d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac"
Oct 30 21:32:29.034089 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.034024    4810 scope.go:122] "RemoveContainer" containerID="d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac"
Oct 30 21:32:29.034893 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:29.034419    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac\": container with ID starting with d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac not found: ID does not exist" containerID="d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac"
Oct 30 21:32:29.034893 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.034533    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac"} err="failed to get container status \"d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac\": rpc error: code = NotFound desc = could not find container \"d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac\": container with ID starting with d27ca6bf860012f2f45c3a271073b9d68eab6acd9fd07be9b368b6ed624a35ac not found: ID does not exist"
Oct 30 21:32:29.050821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.050788    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"]
Oct 30 21:32:29.055806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.055775    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-53aa47/premium-w4-53aa47-kserve-7f99bc6c95-gmg8w"]
Oct 30 21:32:29.171396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.171361    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:29.295115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.295017    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-xks-ca\") pod \"8906e24a-9968-4bcc-895e-6da393ac5ef4\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") "
Oct 30 21:32:29.295115 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.295090    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8906e24a-9968-4bcc-895e-6da393ac5ef4-kube-api-access-6zxs4\" (UniqueName: \"kubernetes.io/projected/8906e24a-9968-4bcc-895e-6da393ac5ef4-kube-api-access-6zxs4\") pod \"8906e24a-9968-4bcc-895e-6da393ac5ef4\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") "
Oct 30 21:32:29.295338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.295169    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-openshift-ca\") pod \"8906e24a-9968-4bcc-895e-6da393ac5ef4\" (UID: \"8906e24a-9968-4bcc-895e-6da393ac5ef4\") "
Oct 30 21:32:29.295526 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.295504    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-openshift-ca" pod "8906e24a-9968-4bcc-895e-6da393ac5ef4" (UID: "8906e24a-9968-4bcc-895e-6da393ac5ef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:29.295621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.295603    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:29.297385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.297356    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-xks-ca" pod "8906e24a-9968-4bcc-895e-6da393ac5ef4" (UID: "8906e24a-9968-4bcc-895e-6da393ac5ef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:29.297524 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.297454    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8906e24a-9968-4bcc-895e-6da393ac5ef4-kube-api-access-6zxs4" pod "8906e24a-9968-4bcc-895e-6da393ac5ef4" (UID: "8906e24a-9968-4bcc-895e-6da393ac5ef4"). InnerVolumeSpecName "kube-api-access-6zxs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:29.396681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.396646    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8906e24a-9968-4bcc-895e-6da393ac5ef4-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:29.396681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.396677    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxs4\" (UniqueName: \"kubernetes.io/projected/8906e24a-9968-4bcc-895e-6da393ac5ef4-kube-api-access-6zxs4\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:29.661801 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.661765    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" path="/var/lib/kubelet/pods/71c1e45e-3f48-47e8-b216-77d2a7d6afca/volumes"
Oct 30 21:32:29.747720 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.747685    4810 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:32:29.747915 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:29.747742    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:32:30.033316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.033276    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z" event={"ID":"8906e24a-9968-4bcc-895e-6da393ac5ef4","Type":"ContainerDied","Data":"6b6317b6030f58549182b4df543c061653fd05cb50992b09005f8defd89b431f"}
Oct 30 21:32:30.033316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.033314    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"
Oct 30 21:32:30.033917 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.033340    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b6317b6030f58549182b4df543c061653fd05cb50992b09005f8defd89b431f"
Oct 30 21:32:30.039161 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.039137    4810 generic.go:356] "Generic (PLEG): container finished" podID="c736a64f-5235-46b3-8981-e4139e7e164c" containerID="344dff5d15669b63be50c58595f1c0ef947a8330051e6b7a2c291ca735e9caf4" exitCode=0
Oct 30 21:32:30.039279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.039170    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" event={"ID":"c736a64f-5235-46b3-8981-e4139e7e164c","Type":"ContainerDied","Data":"344dff5d15669b63be50c58595f1c0ef947a8330051e6b7a2c291ca735e9caf4"}
Oct 30 21:32:30.078913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.078878    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b" podUID="9ac2c41a-2d51-4b93-acd2-55e9271b45ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:32:30.079033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.078890    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb" podUID="0e472de9-60b5-47ae-9889-8540174d0bbf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:32:30.096835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.096812    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:32:30.205463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.205372    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c736a64f-5235-46b3-8981-e4139e7e164c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c736a64f-5235-46b3-8981-e4139e7e164c-istiod-ca-cert\") pod \"c736a64f-5235-46b3-8981-e4139e7e164c\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") "
Oct 30 21:32:30.205463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.205420    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-envoy\") pod \"c736a64f-5235-46b3-8981-e4139e7e164c\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") "
Oct 30 21:32:30.205463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.205450    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c736a64f-5235-46b3-8981-e4139e7e164c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c736a64f-5235-46b3-8981-e4139e7e164c-istio-podinfo\") pod \"c736a64f-5235-46b3-8981-e4139e7e164c\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") "
Oct 30 21:32:30.205769 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.205489    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-certs\") pod \"c736a64f-5235-46b3-8981-e4139e7e164c\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") "
Oct 30 21:32:30.205769 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.205545    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-kube-api-access-lq7ph\" (UniqueName: \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-kube-api-access-lq7ph\") pod \"c736a64f-5235-46b3-8981-e4139e7e164c\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") "
Oct 30 21:32:30.205769 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.205627    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-istio-token\" (UniqueName: \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-istio-token\") pod \"c736a64f-5235-46b3-8981-e4139e7e164c\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") "
Oct 30 21:32:30.205769 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.205715    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-data\") pod \"c736a64f-5235-46b3-8981-e4139e7e164c\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") "
Oct 30 21:32:30.206011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.205850    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-socket\") pod \"c736a64f-5235-46b3-8981-e4139e7e164c\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") "
Oct 30 21:32:30.206011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.205890    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c736a64f-5235-46b3-8981-e4139e7e164c-istiod-ca-cert" pod "c736a64f-5235-46b3-8981-e4139e7e164c" (UID: "c736a64f-5235-46b3-8981-e4139e7e164c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:30.206011 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.205902    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-credential-socket\") pod \"c736a64f-5235-46b3-8981-e4139e7e164c\" (UID: \"c736a64f-5235-46b3-8981-e4139e7e164c\") "
Oct 30 21:32:30.206195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.206168    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-certs" pod "c736a64f-5235-46b3-8981-e4139e7e164c" (UID: "c736a64f-5235-46b3-8981-e4139e7e164c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:30.206257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.206187    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-data" pod "c736a64f-5235-46b3-8981-e4139e7e164c" (UID: "c736a64f-5235-46b3-8981-e4139e7e164c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:30.206340 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.206319    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-socket" pod "c736a64f-5235-46b3-8981-e4139e7e164c" (UID: "c736a64f-5235-46b3-8981-e4139e7e164c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:30.206432 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.206410    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-credential-socket" pod "c736a64f-5235-46b3-8981-e4139e7e164c" (UID: "c736a64f-5235-46b3-8981-e4139e7e164c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:30.206742 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.206721    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.206883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.206745    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.206883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.206761    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.206883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.206775    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.206883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.206789    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c736a64f-5235-46b3-8981-e4139e7e164c-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.208275 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.208250    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c736a64f-5235-46b3-8981-e4139e7e164c-istio-podinfo" pod "c736a64f-5235-46b3-8981-e4139e7e164c" (UID: "c736a64f-5235-46b3-8981-e4139e7e164c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:30.208399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.208379    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-istio-token" pod "c736a64f-5235-46b3-8981-e4139e7e164c" (UID: "c736a64f-5235-46b3-8981-e4139e7e164c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:30.208485 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.208442    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-kube-api-access-lq7ph" pod "c736a64f-5235-46b3-8981-e4139e7e164c" (UID: "c736a64f-5235-46b3-8981-e4139e7e164c"). InnerVolumeSpecName "kube-api-access-lq7ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:30.208538 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.208495    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-envoy" pod "c736a64f-5235-46b3-8981-e4139e7e164c" (UID: "c736a64f-5235-46b3-8981-e4139e7e164c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:30.308511 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.308477    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c736a64f-5235-46b3-8981-e4139e7e164c-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.308511 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.308511    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c736a64f-5235-46b3-8981-e4139e7e164c-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.308764 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.308526    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq7ph\" (UniqueName: \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-kube-api-access-lq7ph\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.308764 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.308539    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c736a64f-5235-46b3-8981-e4139e7e164c-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.433907 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:30.433847    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"]
Oct 30 21:32:31.049892 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.049843    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"
Oct 30 21:32:31.050327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.049890    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9" event={"ID":"c736a64f-5235-46b3-8981-e4139e7e164c","Type":"ContainerDied","Data":"6e93ce2e934adce1e40b37463a84ca60f8fa2182b8afdbc045c37a0dfd05f7e2"}
Oct 30 21:32:31.050327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.049962    4810 scope.go:122] "RemoveContainer" containerID="344dff5d15669b63be50c58595f1c0ef947a8330051e6b7a2c291ca735e9caf4"
Oct 30 21:32:31.079336 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.079312    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"]
Oct 30 21:32:31.087055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.087029    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-7xsn9"]
Oct 30 21:32:31.336820 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.336743    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"]
Oct 30 21:32:31.337410 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337392    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy"
Oct 30 21:32:31.337478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337411    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy"
Oct 30 21:32:31.337478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337444    4810 cpu_manager.go:414] "containerMap: removing container" podUID="49f45b0d-c112-4592-9d24-1f1002eea388" containerName="revoke-keys"
Oct 30 21:32:31.337478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337449    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f45b0d-c112-4592-9d24-1f1002eea388" containerName="revoke-keys"
Oct 30 21:32:31.337478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337456    4810 cpu_manager.go:414] "containerMap: removing container" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main"
Oct 30 21:32:31.337478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337461    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main"
Oct 30 21:32:31.337478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337475    4810 cpu_manager.go:414] "containerMap: removing container" podUID="90687157-a544-4113-aae2-b4c0278794d4" containerName="revoke-keys"
Oct 30 21:32:31.337478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337480    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90687157-a544-4113-aae2-b4c0278794d4" containerName="revoke-keys"
Oct 30 21:32:31.337778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337488    4810 cpu_manager.go:414] "containerMap: removing container" podUID="8906e24a-9968-4bcc-895e-6da393ac5ef4" containerName="revoke-keys"
Oct 30 21:32:31.337778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337493    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8906e24a-9968-4bcc-895e-6da393ac5ef4" containerName="revoke-keys"
Oct 30 21:32:31.337778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337610    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" containerName="istio-proxy"
Oct 30 21:32:31.337778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337619    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f45b0d-c112-4592-9d24-1f1002eea388" containerName="revoke-keys"
Oct 30 21:32:31.337778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337627    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c1e45e-3f48-47e8-b216-77d2a7d6afca" containerName="main"
Oct 30 21:32:31.337778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337634    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="90687157-a544-4113-aae2-b4c0278794d4" containerName="revoke-keys"
Oct 30 21:32:31.337778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.337640    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="8906e24a-9968-4bcc-895e-6da393ac5ef4" containerName="revoke-keys"
Oct 30 21:32:31.342822 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.342782    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.348838 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.348807    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"]
Oct 30 21:32:31.373455 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.373432    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:31.374046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.373621    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:32:31.424665 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.424611    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.426831 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.426789    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vnjs\" (UniqueName: \"kubernetes.io/projected/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-kube-api-access-6vnjs\") pod \"maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.427086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.427064    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.528482 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.528427    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.528683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.528542    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vnjs\" (UniqueName: \"kubernetes.io/projected/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-kube-api-access-6vnjs\") pod \"maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.528683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.528588    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.529459 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.529433    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.531127 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.531092    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.537518 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.537494    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vnjs\" (UniqueName: \"kubernetes.io/projected/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-kube-api-access-6vnjs\") pod \"maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.665025 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.664987    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:31.668083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.668044    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c736a64f-5235-46b3-8981-e4139e7e164c" path="/var/lib/kubelet/pods/c736a64f-5235-46b3-8981-e4139e7e164c/volumes"
Oct 30 21:32:31.956787 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:31.956756    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"]
Oct 30 21:32:31.957986 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:31.957962    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9edee9a_22e5_4e20_85c4_be9d77dcd0bd.slice/crio-57009096171b9aea792bb521ce08a4defbe8ac20e6a8100c65b84add09946264 WatchSource:0}: Error finding container 57009096171b9aea792bb521ce08a4defbe8ac20e6a8100c65b84add09946264: Status 404 returned error can't find the container with id 57009096171b9aea792bb521ce08a4defbe8ac20e6a8100c65b84add09946264
Oct 30 21:32:32.059976 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:32.059947    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs" event={"ID":"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd","Type":"ContainerStarted","Data":"57009096171b9aea792bb521ce08a4defbe8ac20e6a8100c65b84add09946264"}
Oct 30 21:32:32.139631 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:32.139592    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:32:32.844560 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:32.844519    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:32:33.064561 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.064479    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"]
Oct 30 21:32:33.068789 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.068766    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.069577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.069546    4810 generic.go:356] "Generic (PLEG): container finished" podID="a9edee9a-22e5-4e20-85c4-be9d77dcd0bd" containerID="f29c2fd0f9a149fc164145b84ae4053309f4d9fc22579d92afbd0968919af2d8" exitCode=0
Oct 30 21:32:33.069702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.069630    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs" event={"ID":"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd","Type":"ContainerDied","Data":"f29c2fd0f9a149fc164145b84ae4053309f4d9fc22579d92afbd0968919af2d8"}
Oct 30 21:32:33.078661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.078633    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"]
Oct 30 21:32:33.115886 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.115844    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-53aa47-openshift-default-dockercfg-flpcg\""
Oct 30 21:32:33.149612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.149578    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-credential-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.149760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.149666    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176f280b-f3ff-4232-8984-639ab7fb711f-istio-podinfo\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.149760 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.149742    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176f280b-f3ff-4232-8984-639ab7fb711f-istiod-ca-cert\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.149873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.149783    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-data\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.149873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.149808    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.149949 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.149913    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sv76\" (UniqueName: \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-kube-api-access-8sv76\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.150012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.149998    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-certs\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.150050 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.150040    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-envoy\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.150092 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.150078    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-istio-token\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.251283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.251254    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-certs\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.251469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.251294    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-envoy\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.251469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.251333    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-istio-token\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.251469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.251399    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-credential-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.251469 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.251441    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176f280b-f3ff-4232-8984-639ab7fb711f-istio-podinfo\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.251728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.251483    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176f280b-f3ff-4232-8984-639ab7fb711f-istiod-ca-cert\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.251728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.251518    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-data\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.251728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.251685    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.251940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.251774    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sv76\" (UniqueName: \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-kube-api-access-8sv76\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.251940 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.251907    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-data\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.252112 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.252080    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.252195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.252170    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-credential-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.252357 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.252323    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176f280b-f3ff-4232-8984-639ab7fb711f-istiod-ca-cert\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.252453 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.252432    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-certs\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.253800 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.253773    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-envoy\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.254196 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.254168    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176f280b-f3ff-4232-8984-639ab7fb711f-istio-podinfo\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.259692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.259671    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-istio-token\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.260086 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.260061    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sv76\" (UniqueName: \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-kube-api-access-8sv76\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.388064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.388033    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:33.549045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.545946    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"]
Oct 30 21:32:33.549045 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.546228    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerName="istio-proxy" containerID="cri-o://df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0" gracePeriod=30
Oct 30 21:32:33.688602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.688449    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"]
Oct 30 21:32:33.690742 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:33.690710    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176f280b_f3ff_4232_8984_639ab7fb711f.slice/crio-f863eeea10b88ed32a47ab25e6b0a0312c3ad33e6aa9ec0e094d8855a6bd1078 WatchSource:0}: Error finding container f863eeea10b88ed32a47ab25e6b0a0312c3ad33e6aa9ec0e094d8855a6bd1078: Status 404 returned error can't find the container with id f863eeea10b88ed32a47ab25e6b0a0312c3ad33e6aa9ec0e094d8855a6bd1078
Oct 30 21:32:33.693902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.693826    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:33.693980 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.693961    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:33.694027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:33.694009    4810 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:34.039495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.039402    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"]
Oct 30 21:32:34.043826 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.043801    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.055986 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.055955    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"]
Oct 30 21:32:34.081685 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.081641    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" event={"ID":"176f280b-f3ff-4232-8984-639ab7fb711f","Type":"ContainerStarted","Data":"5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1"}
Oct 30 21:32:34.082205 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.081704    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" event={"ID":"176f280b-f3ff-4232-8984-639ab7fb711f","Type":"ContainerStarted","Data":"f863eeea10b88ed32a47ab25e6b0a0312c3ad33e6aa9ec0e094d8855a6bd1078"}
Oct 30 21:32:34.105490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.105419    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" podStartSLOduration=1.105400239 podStartE2EDuration="1.105400239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:34.103213602 +0000 UTC m=+3320.595030147" watchObservedRunningTime="2026-10-30 21:32:34.105400239 +0000 UTC m=+3320.597216786"
Oct 30 21:32:34.174345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.172844    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.174345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.173016    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.174345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.173042    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8qj\" (UniqueName: \"kubernetes.io/projected/dfafe974-e665-48d1-b7e3-0d090e49f2fd-kube-api-access-ws8qj\") pod \"maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.244436 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.244408    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:34.274251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.274214    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.274508 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.274485    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8qj\" (UniqueName: \"kubernetes.io/projected/dfafe974-e665-48d1-b7e3-0d090e49f2fd-kube-api-access-ws8qj\") pod \"maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.274996 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.274934    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.276614 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.276585    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.283876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.279788    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.313168 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.313082    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8qj\" (UniqueName: \"kubernetes.io/projected/dfafe974-e665-48d1-b7e3-0d090e49f2fd-kube-api-access-ws8qj\") pod \"maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.360185 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.360156    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:34.379885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.377244    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-xks-ca\") pod \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") "
Oct 30 21:32:34.379885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.377462    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-openshift-ca\") pod \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") "
Oct 30 21:32:34.379885 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.377576    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-kube-api-access-6vnjs\" (UniqueName: \"kubernetes.io/projected/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-kube-api-access-6vnjs\") pod \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\" (UID: \"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd\") "
Oct 30 21:32:34.384887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.380515    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-openshift-ca" pod "a9edee9a-22e5-4e20-85c4-be9d77dcd0bd" (UID: "a9edee9a-22e5-4e20-85c4-be9d77dcd0bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:34.400823 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.400270    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:34.400823 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.400463    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-xks-ca" pod "a9edee9a-22e5-4e20-85c4-be9d77dcd0bd" (UID: "a9edee9a-22e5-4e20-85c4-be9d77dcd0bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:34.401462 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.401429    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-kube-api-access-6vnjs" pod "a9edee9a-22e5-4e20-85c4-be9d77dcd0bd" (UID: "a9edee9a-22e5-4e20-85c4-be9d77dcd0bd"). InnerVolumeSpecName "kube-api-access-6vnjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:34.402446 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.402415    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 21:32:34.402569 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.402464    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 21:32:34.491880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.490451    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vnjs\" (UniqueName: \"kubernetes.io/projected/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-kube-api-access-6vnjs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:34.491880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.490492    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:34.491880 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.490508    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:34.670959 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:34.670925    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"]
Oct 30 21:32:35.103922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:35.103847    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs" event={"ID":"a9edee9a-22e5-4e20-85c4-be9d77dcd0bd","Type":"ContainerDied","Data":"57009096171b9aea792bb521ce08a4defbe8ac20e6a8100c65b84add09946264"}
Oct 30 21:32:35.104434 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:35.103947    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57009096171b9aea792bb521ce08a4defbe8ac20e6a8100c65b84add09946264"
Oct 30 21:32:35.104434 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:35.104031    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"
Oct 30 21:32:35.106891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:35.106853    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj" event={"ID":"dfafe974-e665-48d1-b7e3-0d090e49f2fd","Type":"ContainerStarted","Data":"eff953f661f90225b65e5acd78ec166f88f5a6a210fa42be2e5012630e3c7b76"}
Oct 30 21:32:35.390071 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:35.390034    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 21:32:35.390338 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:35.390312    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 21:32:35.726109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:35.726025    4810 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:32:35.726109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:35.726088    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:32:36.141969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:36.141930    4810 generic.go:356] "Generic (PLEG): container finished" podID="dfafe974-e665-48d1-b7e3-0d090e49f2fd" containerID="d578c352fe31f2d7e40d1379655e6c57a343b8f68ed67b5f2fb66c877888879b" exitCode=0
Oct 30 21:32:36.142946 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:36.141991    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj" event={"ID":"dfafe974-e665-48d1-b7e3-0d090e49f2fd","Type":"ContainerDied","Data":"d578c352fe31f2d7e40d1379655e6c57a343b8f68ed67b5f2fb66c877888879b"}
Oct 30 21:32:36.392509 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:36.392417    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 21:32:36.392759 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:36.392728    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 21:32:37.393402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.392277    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 21:32:37.393402 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.392341    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 21:32:37.454460 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.454427    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:37.591878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.577723    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-xks-ca\") pod \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") "
Oct 30 21:32:37.591878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.577785    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-openshift-ca\") pod \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") "
Oct 30 21:32:37.591878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.577988    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfafe974-e665-48d1-b7e3-0d090e49f2fd-kube-api-access-ws8qj\" (UniqueName: \"kubernetes.io/projected/dfafe974-e665-48d1-b7e3-0d090e49f2fd-kube-api-access-ws8qj\") pod \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\" (UID: \"dfafe974-e665-48d1-b7e3-0d090e49f2fd\") "
Oct 30 21:32:37.591878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.584715    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-openshift-ca" pod "dfafe974-e665-48d1-b7e3-0d090e49f2fd" (UID: "dfafe974-e665-48d1-b7e3-0d090e49f2fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:37.591878 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.588017    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-xks-ca" pod "dfafe974-e665-48d1-b7e3-0d090e49f2fd" (UID: "dfafe974-e665-48d1-b7e3-0d090e49f2fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:37.614094 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.601031    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfafe974-e665-48d1-b7e3-0d090e49f2fd-kube-api-access-ws8qj" pod "dfafe974-e665-48d1-b7e3-0d090e49f2fd" (UID: "dfafe974-e665-48d1-b7e3-0d090e49f2fd"). InnerVolumeSpecName "kube-api-access-ws8qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:37.682388 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.681774    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.682388 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.681815    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfafe974-e665-48d1-b7e3-0d090e49f2fd-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.682388 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:37.681831    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8qj\" (UniqueName: \"kubernetes.io/projected/dfafe974-e665-48d1-b7e3-0d090e49f2fd-kube-api-access-ws8qj\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:38.169876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.169833    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"
Oct 30 21:32:38.169876 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.169847    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj" event={"ID":"dfafe974-e665-48d1-b7e3-0d090e49f2fd","Type":"ContainerDied","Data":"eff953f661f90225b65e5acd78ec166f88f5a6a210fa42be2e5012630e3c7b76"}
Oct 30 21:32:38.170145 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.169916    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eff953f661f90225b65e5acd78ec166f88f5a6a210fa42be2e5012630e3c7b76"
Oct 30 21:32:38.263304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.262473    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"]
Oct 30 21:32:38.263304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.263093    4810 cpu_manager.go:414] "containerMap: removing container" podUID="dfafe974-e665-48d1-b7e3-0d090e49f2fd" containerName="revoke-keys"
Oct 30 21:32:38.263304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.263108    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfafe974-e665-48d1-b7e3-0d090e49f2fd" containerName="revoke-keys"
Oct 30 21:32:38.263304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.263127    4810 cpu_manager.go:414] "containerMap: removing container" podUID="a9edee9a-22e5-4e20-85c4-be9d77dcd0bd" containerName="revoke-keys"
Oct 30 21:32:38.263304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.263132    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9edee9a-22e5-4e20-85c4-be9d77dcd0bd" containerName="revoke-keys"
Oct 30 21:32:38.263304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.263222    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9edee9a-22e5-4e20-85c4-be9d77dcd0bd" containerName="revoke-keys"
Oct 30 21:32:38.263304 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.263239    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfafe974-e665-48d1-b7e3-0d090e49f2fd" containerName="revoke-keys"
Oct 30 21:32:38.268018 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.267164    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.269596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.269567    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"]
Oct 30 21:32:38.287881 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.285236    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.309884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.297645    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:38.309884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.299623    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52tt\" (UniqueName: \"kubernetes.io/projected/33b09af8-6024-4169-82ea-98697ee8265f-kube-api-access-g52tt\") pod \"maas-api-revoke-sub-02aabb1d-rvxkb\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.309884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.299745    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02aabb1d-rvxkb\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.309884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.299786    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02aabb1d-rvxkb\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.309884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.299823    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd5ac09-nvzxg\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.309884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.299905    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd5ac09-nvzxg\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.309884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.299961    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gpj\" (UniqueName: \"kubernetes.io/projected/2ba93cab-0db4-4f3f-869a-967924134e51-kube-api-access-j8gpj\") pod \"maas-api-revoke-sub-bfd5ac09-nvzxg\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.309884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.301050    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:32:38.337053 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.337018    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"]
Oct 30 21:32:38.349374 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.349338    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"]
Oct 30 21:32:38.422933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.422040    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52tt\" (UniqueName: \"kubernetes.io/projected/33b09af8-6024-4169-82ea-98697ee8265f-kube-api-access-g52tt\") pod \"maas-api-revoke-sub-02aabb1d-rvxkb\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.433607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.433113    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02aabb1d-rvxkb\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.433607 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.433187    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02aabb1d-rvxkb\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.438934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.434132    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02aabb1d-rvxkb\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.438934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.434687    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd5ac09-nvzxg\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.438934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.434765    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd5ac09-nvzxg\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.438934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.434822    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gpj\" (UniqueName: \"kubernetes.io/projected/2ba93cab-0db4-4f3f-869a-967924134e51-kube-api-access-j8gpj\") pod \"maas-api-revoke-sub-bfd5ac09-nvzxg\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.438934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.435517    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd5ac09-nvzxg\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.451136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.450511    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd5ac09-nvzxg\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.451136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.450531    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02aabb1d-rvxkb\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.451136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.450618    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 21:32:38.451136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.450663    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 21:32:38.454757 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.454631    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gpj\" (UniqueName: \"kubernetes.io/projected/2ba93cab-0db4-4f3f-869a-967924134e51-kube-api-access-j8gpj\") pod \"maas-api-revoke-sub-bfd5ac09-nvzxg\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.454891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.454784    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52tt\" (UniqueName: \"kubernetes.io/projected/33b09af8-6024-4169-82ea-98697ee8265f-kube-api-access-g52tt\") pod \"maas-api-revoke-sub-02aabb1d-rvxkb\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.654612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.654578    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:38.670083 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.669439    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:38.717046 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.717004    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"]
Oct 30 21:32:38.717296 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:38.717284    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-53aa47-dkdph"]
Oct 30 21:32:39.036579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.036549    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:32:39.114956 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.114828    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"]
Oct 30 21:32:39.115793 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:39.115753    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b09af8_6024_4169_82ea_98697ee8265f.slice/crio-47c6a5be4d5d95c808f56fc6d3a4cc14ea75007765ba364ac1b7425123d04400 WatchSource:0}: Error finding container 47c6a5be4d5d95c808f56fc6d3a4cc14ea75007765ba364ac1b7425123d04400: Status 404 returned error can't find the container with id 47c6a5be4d5d95c808f56fc6d3a4cc14ea75007765ba364ac1b7425123d04400
Oct 30 21:32:39.133417 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.133373    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"]
Oct 30 21:32:39.154313 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154082    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-credential-socket\") pod \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") "
Oct 30 21:32:39.154313 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154133    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-podinfo\") pod \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") "
Oct 30 21:32:39.154313 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154168    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-socket\") pod \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") "
Oct 30 21:32:39.154313 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154196    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-token\" (UniqueName: \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-token\") pod \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") "
Oct 30 21:32:39.154608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154395    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-envoy\") pod \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") "
Oct 30 21:32:39.154608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154504    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-certs\") pod \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") "
Oct 30 21:32:39.154723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154620    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-kube-api-access-pjdh8\" (UniqueName: \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-kube-api-access-pjdh8\") pod \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") "
Oct 30 21:32:39.154723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154674    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istiod-ca-cert\") pod \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") "
Oct 30 21:32:39.154723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154709    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-data\") pod \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\" (UID: \"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7\") "
Oct 30 21:32:39.154947 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154778    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-socket" pod "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" (UID: "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:39.155015 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.154957    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-certs" pod "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" (UID: "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:39.155163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.155131    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-data" pod "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" (UID: "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:39.155238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.155156    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istiod-ca-cert" pod "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" (UID: "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:39.155238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.155159    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.155238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.155201    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.155809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.155787    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-credential-socket" pod "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" (UID: "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:39.157596 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.157570    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-podinfo" pod "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" (UID: "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:39.157700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.157611    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-kube-api-access-pjdh8" pod "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" (UID: "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7"). InnerVolumeSpecName "kube-api-access-pjdh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:39.157902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.157879    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-envoy" pod "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" (UID: "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:39.159585 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.159562    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-token" pod "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" (UID: "9c8e714d-9fef-4dd0-8bdb-62059aeb0be7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:39.211993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.211951    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb" event={"ID":"33b09af8-6024-4169-82ea-98697ee8265f","Type":"ContainerStarted","Data":"47c6a5be4d5d95c808f56fc6d3a4cc14ea75007765ba364ac1b7425123d04400"}
Oct 30 21:32:39.214174 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.214118    4810 generic.go:356] "Generic (PLEG): container finished" podID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerID="df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0" exitCode=0
Oct 30 21:32:39.214287 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.214276    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"
Oct 30 21:32:39.216257 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.216218    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" event={"ID":"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7","Type":"ContainerDied","Data":"df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0"}
Oct 30 21:32:39.216372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.216279    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb" event={"ID":"9c8e714d-9fef-4dd0-8bdb-62059aeb0be7","Type":"ContainerDied","Data":"7b50326c30bd633bce9361412d67ba591b44ba4bd8d5dc25617838e0b4c71bbf"}
Oct 30 21:32:39.216372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.216311    4810 scope.go:122] "RemoveContainer" containerID="df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0"
Oct 30 21:32:39.217644 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.217615    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg" event={"ID":"2ba93cab-0db4-4f3f-869a-967924134e51","Type":"ContainerStarted","Data":"00f5d13ce9dd80c7359a8dd9ca291d96a6193b02ddad85a0d5a205d9760580d2"}
Oct 30 21:32:39.231076 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.231056    4810 scope.go:122] "RemoveContainer" containerID="df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0"
Oct 30 21:32:39.231349 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:39.231330    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0\": container with ID starting with df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0 not found: ID does not exist" containerID="df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0"
Oct 30 21:32:39.231400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.231359    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0"} err="failed to get container status \"df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0\": rpc error: code = NotFound desc = could not find container \"df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0\": container with ID starting with df5d3a4bcdaba9a48dfb8bd1509f4db71aa52d21fcc621fd7b730db5d8ebe4d0 not found: ID does not exist"
Oct 30 21:32:39.241467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.241443    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"]
Oct 30 21:32:39.247548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.247529    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-qbdlb"]
Oct 30 21:32:39.256487 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.256466    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjdh8\" (UniqueName: \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-kube-api-access-pjdh8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.256600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.256589    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.256670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.256662    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.256726 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.256719    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.256781 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.256773    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.256868 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.256836    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.256938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.256899    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.391345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.391307    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 21:32:39.392556 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.392524    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 21:32:39.663372 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.663339    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eef0c04-e32d-46f5-b691-5b7256b84214" path="/var/lib/kubelet/pods/7eef0c04-e32d-46f5-b691-5b7256b84214/volumes"
Oct 30 21:32:39.663814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:39.663797    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" path="/var/lib/kubelet/pods/9c8e714d-9fef-4dd0-8bdb-62059aeb0be7/volumes"
Oct 30 21:32:40.081912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:40.081733    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb" podUID="0e472de9-60b5-47ae-9889-8540174d0bbf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:32:40.082592 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:40.082562    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b" podUID="9ac2c41a-2d51-4b93-acd2-55e9271b45ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:32:40.229355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:40.229307    4810 generic.go:356] "Generic (PLEG): container finished" podID="2ba93cab-0db4-4f3f-869a-967924134e51" containerID="cd16edff36543eae9ecd522743b6cd19134d57ec7292be2824f1ffc0b1cdb682" exitCode=0
Oct 30 21:32:40.229575 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:40.229383    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg" event={"ID":"2ba93cab-0db4-4f3f-869a-967924134e51","Type":"ContainerDied","Data":"cd16edff36543eae9ecd522743b6cd19134d57ec7292be2824f1ffc0b1cdb682"}
Oct 30 21:32:40.231347 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:40.231317    4810 generic.go:356] "Generic (PLEG): container finished" podID="33b09af8-6024-4169-82ea-98697ee8265f" containerID="57a3c991cec4d218a54d5f1c4a565f8591cbd368de6786a50a0cc386e3d526d4" exitCode=0
Oct 30 21:32:40.231498 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:40.231403    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb" event={"ID":"33b09af8-6024-4169-82ea-98697ee8265f","Type":"ContainerDied","Data":"57a3c991cec4d218a54d5f1c4a565f8591cbd368de6786a50a0cc386e3d526d4"}
Oct 30 21:32:40.389186 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:40.389149    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused" start-of-body=
Oct 30 21:32:40.389389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:40.389217    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": dial tcp 10.132.1.178:15021: connect: connection refused"
Oct 30 21:32:41.555570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.555438    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:41.565545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.561028    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:41.693510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.690402    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b09af8-6024-4169-82ea-98697ee8265f-kube-api-access-g52tt\" (UniqueName: \"kubernetes.io/projected/33b09af8-6024-4169-82ea-98697ee8265f-kube-api-access-g52tt\") pod \"33b09af8-6024-4169-82ea-98697ee8265f\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") "
Oct 30 21:32:41.693510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.690469    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-xks-ca\") pod \"2ba93cab-0db4-4f3f-869a-967924134e51\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") "
Oct 30 21:32:41.693510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.690520    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-xks-ca\") pod \"33b09af8-6024-4169-82ea-98697ee8265f\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") "
Oct 30 21:32:41.693510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.690606    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-openshift-ca\") pod \"33b09af8-6024-4169-82ea-98697ee8265f\" (UID: \"33b09af8-6024-4169-82ea-98697ee8265f\") "
Oct 30 21:32:41.693510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.690636    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba93cab-0db4-4f3f-869a-967924134e51-kube-api-access-j8gpj\" (UniqueName: \"kubernetes.io/projected/2ba93cab-0db4-4f3f-869a-967924134e51-kube-api-access-j8gpj\") pod \"2ba93cab-0db4-4f3f-869a-967924134e51\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") "
Oct 30 21:32:41.693510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.690737    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-openshift-ca\") pod \"2ba93cab-0db4-4f3f-869a-967924134e51\" (UID: \"2ba93cab-0db4-4f3f-869a-967924134e51\") "
Oct 30 21:32:41.693510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.691541    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-openshift-ca" pod "2ba93cab-0db4-4f3f-869a-967924134e51" (UID: "2ba93cab-0db4-4f3f-869a-967924134e51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:41.695345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.694690    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-openshift-ca" pod "33b09af8-6024-4169-82ea-98697ee8265f" (UID: "33b09af8-6024-4169-82ea-98697ee8265f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:41.706358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.705687    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b09af8-6024-4169-82ea-98697ee8265f-kube-api-access-g52tt" pod "33b09af8-6024-4169-82ea-98697ee8265f" (UID: "33b09af8-6024-4169-82ea-98697ee8265f"). InnerVolumeSpecName "kube-api-access-g52tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:41.710197 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.710119    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-xks-ca" pod "2ba93cab-0db4-4f3f-869a-967924134e51" (UID: "2ba93cab-0db4-4f3f-869a-967924134e51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:41.712801 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.712004    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba93cab-0db4-4f3f-869a-967924134e51-kube-api-access-j8gpj" pod "2ba93cab-0db4-4f3f-869a-967924134e51" (UID: "2ba93cab-0db4-4f3f-869a-967924134e51"). InnerVolumeSpecName "kube-api-access-j8gpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:41.720145 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.719705    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:32:41.727136 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.727102    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-xks-ca" pod "33b09af8-6024-4169-82ea-98697ee8265f" (UID: "33b09af8-6024-4169-82ea-98697ee8265f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:41.805993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.796268    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e472de9-60b5-47ae-9889-8540174d0bbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e472de9-60b5-47ae-9889-8540174d0bbf-plugins-config-volume\") pod \"0e472de9-60b5-47ae-9889-8540174d0bbf\" (UID: \"0e472de9-60b5-47ae-9889-8540174d0bbf\") "
Oct 30 21:32:41.805993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.796507    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e472de9-60b5-47ae-9889-8540174d0bbf-kube-api-access-wsn9q\" (UniqueName: \"kubernetes.io/projected/0e472de9-60b5-47ae-9889-8540174d0bbf-kube-api-access-wsn9q\") pod \"0e472de9-60b5-47ae-9889-8540174d0bbf\" (UID: \"0e472de9-60b5-47ae-9889-8540174d0bbf\") "
Oct 30 21:32:41.837912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.811351    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:41.837912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.811389    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g52tt\" (UniqueName: \"kubernetes.io/projected/33b09af8-6024-4169-82ea-98697ee8265f-kube-api-access-g52tt\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:41.837912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.811405    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba93cab-0db4-4f3f-869a-967924134e51-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:41.837912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.811420    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:41.837912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.811434    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b09af8-6024-4169-82ea-98697ee8265f-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:41.837912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.811450    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8gpj\" (UniqueName: \"kubernetes.io/projected/2ba93cab-0db4-4f3f-869a-967924134e51-kube-api-access-j8gpj\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:41.837912 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.822364    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e472de9-60b5-47ae-9889-8540174d0bbf-kube-api-access-wsn9q" pod "0e472de9-60b5-47ae-9889-8540174d0bbf" (UID: "0e472de9-60b5-47ae-9889-8540174d0bbf"). InnerVolumeSpecName "kube-api-access-wsn9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:41.838896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.838818    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e472de9-60b5-47ae-9889-8540174d0bbf-plugins-config-volume" pod "0e472de9-60b5-47ae-9889-8540174d0bbf" (UID: "0e472de9-60b5-47ae-9889-8540174d0bbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:41.841390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.841035    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:32:41.913210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.912378    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-kube-api-access-97bsl\" (UniqueName: \"kubernetes.io/projected/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-kube-api-access-97bsl\") pod \"9ac2c41a-2d51-4b93-acd2-55e9271b45ab\" (UID: \"9ac2c41a-2d51-4b93-acd2-55e9271b45ab\") "
Oct 30 21:32:41.913210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.912490    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-plugins-config-volume\") pod \"9ac2c41a-2d51-4b93-acd2-55e9271b45ab\" (UID: \"9ac2c41a-2d51-4b93-acd2-55e9271b45ab\") "
Oct 30 21:32:41.913210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.912922    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e472de9-60b5-47ae-9889-8540174d0bbf-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:41.913210 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.912947    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsn9q\" (UniqueName: \"kubernetes.io/projected/0e472de9-60b5-47ae-9889-8540174d0bbf-kube-api-access-wsn9q\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:41.914148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.913782    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-plugins-config-volume" pod "9ac2c41a-2d51-4b93-acd2-55e9271b45ab" (UID: "9ac2c41a-2d51-4b93-acd2-55e9271b45ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:41.918427 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:41.918387    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-kube-api-access-97bsl" pod "9ac2c41a-2d51-4b93-acd2-55e9271b45ab" (UID: "9ac2c41a-2d51-4b93-acd2-55e9271b45ab"). InnerVolumeSpecName "kube-api-access-97bsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:42.014652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.014543    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.014652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.014579    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97bsl\" (UniqueName: \"kubernetes.io/projected/9ac2c41a-2d51-4b93-acd2-55e9271b45ab-kube-api-access-97bsl\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.247200 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.247162    4810 generic.go:356] "Generic (PLEG): container finished" podID="9ac2c41a-2d51-4b93-acd2-55e9271b45ab" containerID="f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c" exitCode=0
Oct 30 21:32:42.247392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.247212    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"
Oct 30 21:32:42.247392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.247245    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b" event={"ID":"9ac2c41a-2d51-4b93-acd2-55e9271b45ab","Type":"ContainerDied","Data":"f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c"}
Oct 30 21:32:42.247392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.247311    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b" event={"ID":"9ac2c41a-2d51-4b93-acd2-55e9271b45ab","Type":"ContainerDied","Data":"b6508ad8664b8fcabeaa4d51e7fd6705bc72b010a9175e209f89d6160f72cf58"}
Oct 30 21:32:42.247392 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.247337    4810 scope.go:122] "RemoveContainer" containerID="f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c"
Oct 30 21:32:42.249180 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.249153    4810 generic.go:356] "Generic (PLEG): container finished" podID="0e472de9-60b5-47ae-9889-8540174d0bbf" containerID="6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27" exitCode=0
Oct 30 21:32:42.249316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.249204    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"
Oct 30 21:32:42.249316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.249254    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb" event={"ID":"0e472de9-60b5-47ae-9889-8540174d0bbf","Type":"ContainerDied","Data":"6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27"}
Oct 30 21:32:42.249899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.249840    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb" event={"ID":"0e472de9-60b5-47ae-9889-8540174d0bbf","Type":"ContainerDied","Data":"96445c54e405de232916a8e45bbc1cf6fea2cfbf378b84f624857c1ec612b311"}
Oct 30 21:32:42.251099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.251016    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg"
Oct 30 21:32:42.251099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.251036    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd5ac09-nvzxg" event={"ID":"2ba93cab-0db4-4f3f-869a-967924134e51","Type":"ContainerDied","Data":"00f5d13ce9dd80c7359a8dd9ca291d96a6193b02ddad85a0d5a205d9760580d2"}
Oct 30 21:32:42.251099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.251082    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00f5d13ce9dd80c7359a8dd9ca291d96a6193b02ddad85a0d5a205d9760580d2"
Oct 30 21:32:42.252751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.252661    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb" event={"ID":"33b09af8-6024-4169-82ea-98697ee8265f","Type":"ContainerDied","Data":"47c6a5be4d5d95c808f56fc6d3a4cc14ea75007765ba364ac1b7425123d04400"}
Oct 30 21:32:42.252751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.252699    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47c6a5be4d5d95c808f56fc6d3a4cc14ea75007765ba364ac1b7425123d04400"
Oct 30 21:32:42.252751 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.252751    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02aabb1d-rvxkb"
Oct 30 21:32:42.261435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.261417    4810 scope.go:122] "RemoveContainer" containerID="f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c"
Oct 30 21:32:42.262239 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:42.262179    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c\": container with ID starting with f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c not found: ID does not exist" containerID="f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c"
Oct 30 21:32:42.262239 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.262218    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c"} err="failed to get container status \"f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c\": rpc error: code = NotFound desc = could not find container \"f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c\": container with ID starting with f08517ddd1ac656ca0c7fb6ad5b8b29354ca16a0cfc0cf64d3ad6b2b4631a37c not found: ID does not exist"
Oct 30 21:32:42.262385 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.262242    4810 scope.go:122] "RemoveContainer" containerID="6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27"
Oct 30 21:32:42.279604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.278774    4810 scope.go:122] "RemoveContainer" containerID="6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27"
Oct 30 21:32:42.279604 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:42.279505    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27\": container with ID starting with 6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27 not found: ID does not exist" containerID="6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27"
Oct 30 21:32:42.279604 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.279542    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27"} err="failed to get container status \"6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27\": rpc error: code = NotFound desc = could not find container \"6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27\": container with ID starting with 6342907e99e8ea0051770831b08155f3e89b8f5e509aa71d15522334b9fc9a27 not found: ID does not exist"
Oct 30 21:32:42.283799 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.283752    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"]
Oct 30 21:32:42.295004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.294948    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f64640-9df885d47-qvq5b"]
Oct 30 21:32:42.349989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.346572    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"]
Oct 30 21:32:42.350723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.350703    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f64640-6f868446cc-q45qb"]
Oct 30 21:32:42.431066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.431023    4810 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:32:42.431244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.431095    4810 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:42.499330 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.499292    4810 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:42.499661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.499641    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:42.501149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.501124    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:42.598969 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:42.598834    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:32:42.917685 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:42.917646    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:32:43.661630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:43.661594    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e472de9-60b5-47ae-9889-8540174d0bbf" path="/var/lib/kubelet/pods/0e472de9-60b5-47ae-9889-8540174d0bbf/volumes"
Oct 30 21:32:43.662064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:43.662035    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac2c41a-2d51-4b93-acd2-55e9271b45ab" path="/var/lib/kubelet/pods/9ac2c41a-2d51-4b93-acd2-55e9271b45ab/volumes"
Oct 30 21:32:45.457452 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.457414    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" containerID="cri-o://4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321" gracePeriod=45
Oct 30 21:32:45.630771 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.630729    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 30 21:32:45.754974 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.754947    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 21:32:45.873681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.871383    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-home\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-home\") pod \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") "
Oct 30 21:32:45.873681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.871559    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tmp-dir\") pod \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") "
Oct 30 21:32:45.873681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.871621    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abdc7e8-e62f-455f-94b4-603fb02a88f8-kube-api-access-6vslm\" (UniqueName: \"kubernetes.io/projected/9abdc7e8-e62f-455f-94b4-603fb02a88f8-kube-api-access-6vslm\") pod \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") "
Oct 30 21:32:45.873681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.871650    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-dshm\") pod \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") "
Oct 30 21:32:45.873681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.871675    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tls-certs\") pod \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") "
Oct 30 21:32:45.873681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.871734    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-model-cache\") pod \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\" (UID: \"9abdc7e8-e62f-455f-94b4-603fb02a88f8\") "
Oct 30 21:32:45.873681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.871833    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tmp-dir" pod "9abdc7e8-e62f-455f-94b4-603fb02a88f8" (UID: "9abdc7e8-e62f-455f-94b4-603fb02a88f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:45.873681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.872146    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-home" pod "9abdc7e8-e62f-455f-94b4-603fb02a88f8" (UID: "9abdc7e8-e62f-455f-94b4-603fb02a88f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:45.873681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.872333    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-model-cache" pod "9abdc7e8-e62f-455f-94b4-603fb02a88f8" (UID: "9abdc7e8-e62f-455f-94b4-603fb02a88f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:45.874396 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.874311    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abdc7e8-e62f-455f-94b4-603fb02a88f8-kube-api-access-6vslm" pod "9abdc7e8-e62f-455f-94b4-603fb02a88f8" (UID: "9abdc7e8-e62f-455f-94b4-603fb02a88f8"). InnerVolumeSpecName "kube-api-access-6vslm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:45.874728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.874690    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:45.874728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.874721    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vslm\" (UniqueName: \"kubernetes.io/projected/9abdc7e8-e62f-455f-94b4-603fb02a88f8-kube-api-access-6vslm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:45.874728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.874736    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:45.875055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.874749    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:45.875710 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.875685    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tls-certs" pod "9abdc7e8-e62f-455f-94b4-603fb02a88f8" (UID: "9abdc7e8-e62f-455f-94b4-603fb02a88f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:45.875808 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.875688    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-dshm" pod "9abdc7e8-e62f-455f-94b4-603fb02a88f8" (UID: "9abdc7e8-e62f-455f-94b4-603fb02a88f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:45.975548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.975459    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abdc7e8-e62f-455f-94b4-603fb02a88f8-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:45.975548 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:45.975489    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abdc7e8-e62f-455f-94b4-603fb02a88f8-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:46.297097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.297007    4810 generic.go:356] "Generic (PLEG): container finished" podID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerID="4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321" exitCode=0
Oct 30 21:32:46.297368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.297085    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"
Oct 30 21:32:46.297467 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.297103    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" event={"ID":"9abdc7e8-e62f-455f-94b4-603fb02a88f8","Type":"ContainerDied","Data":"4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321"}
Oct 30 21:32:46.297521 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.297485    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png" event={"ID":"9abdc7e8-e62f-455f-94b4-603fb02a88f8","Type":"ContainerDied","Data":"913a37ac5af59d5d18e739f43493e1e046c1bc4c3d28fb42d8541b4ca0e948c4"}
Oct 30 21:32:46.297570 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.297517    4810 scope.go:122] "RemoveContainer" containerID="4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321"
Oct 30 21:32:46.319026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.319002    4810 scope.go:122] "RemoveContainer" containerID="4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321"
Oct 30 21:32:46.319379 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:46.319344    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321\": container with ID starting with 4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321 not found: ID does not exist" containerID="4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321"
Oct 30 21:32:46.319458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.319390    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321"} err="failed to get container status \"4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321\": rpc error: code = NotFound desc = could not find container \"4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321\": container with ID starting with 4da6f47407e5a7518d221f891871db933724d62c431a6bcd7076873620d9e321 not found: ID does not exist"
Oct 30 21:32:46.331615 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.331577    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"]
Oct 30 21:32:46.335754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.335712    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/premium-w0-bc102e-kserve-7cdc978d6c-c6png"]
Oct 30 21:32:46.857976 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.857941    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"]
Oct 30 21:32:46.858495 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858481    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9ac2c41a-2d51-4b93-acd2-55e9271b45ab" containerName="payload-pre-processing"
Oct 30 21:32:46.858558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858496    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac2c41a-2d51-4b93-acd2-55e9271b45ab" containerName="payload-pre-processing"
Oct 30 21:32:46.858558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858513    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main"
Oct 30 21:32:46.858558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858520    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main"
Oct 30 21:32:46.858558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858545    4810 cpu_manager.go:414] "containerMap: removing container" podUID="33b09af8-6024-4169-82ea-98697ee8265f" containerName="revoke-keys"
Oct 30 21:32:46.858558 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858550    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b09af8-6024-4169-82ea-98697ee8265f" containerName="revoke-keys"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858565    4810 cpu_manager.go:414] "containerMap: removing container" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerName="istio-proxy"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858570    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerName="istio-proxy"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858578    4810 cpu_manager.go:414] "containerMap: removing container" podUID="0e472de9-60b5-47ae-9889-8540174d0bbf" containerName="payload-processing"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858582    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e472de9-60b5-47ae-9889-8540174d0bbf" containerName="payload-processing"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858591    4810 cpu_manager.go:414] "containerMap: removing container" podUID="2ba93cab-0db4-4f3f-869a-967924134e51" containerName="revoke-keys"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858595    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba93cab-0db4-4f3f-869a-967924134e51" containerName="revoke-keys"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858667    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" containerName="main"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858680    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b09af8-6024-4169-82ea-98697ee8265f" containerName="revoke-keys"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858689    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8e714d-9fef-4dd0-8bdb-62059aeb0be7" containerName="istio-proxy"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858695    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e472de9-60b5-47ae-9889-8540174d0bbf" containerName="payload-processing"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858701    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba93cab-0db4-4f3f-869a-967924134e51" containerName="revoke-keys"
Oct 30 21:32:46.858739 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.858708    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac2c41a-2d51-4b93-acd2-55e9271b45ab" containerName="payload-pre-processing"
Oct 30 21:32:46.865103 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.865069    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:46.874319 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.874268    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:46.874493 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.874394    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:32:46.882106 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:46.882082    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"]
Oct 30 21:32:47.007888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.004441    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1a5c3d-7598d\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:47.007888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.004489    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1a5c3d-7598d\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:47.007888 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.005374    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvl9\" (UniqueName: \"kubernetes.io/projected/e8954104-beb7-4d35-96bb-f71f0df8e873-kube-api-access-zhvl9\") pod \"maas-api-revoke-sub-0d1a5c3d-7598d\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:47.008230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.008195    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"]
Oct 30 21:32:47.009274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.008440    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy" containerID="cri-o://5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1" gracePeriod=30
Oct 30 21:32:47.107198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.107152    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvl9\" (UniqueName: \"kubernetes.io/projected/e8954104-beb7-4d35-96bb-f71f0df8e873-kube-api-access-zhvl9\") pod \"maas-api-revoke-sub-0d1a5c3d-7598d\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:47.107368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.107291    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1a5c3d-7598d\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:47.107368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.107325    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1a5c3d-7598d\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:47.108312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.108245    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1a5c3d-7598d\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:47.109910 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.109884    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1a5c3d-7598d\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:47.119366 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.119336    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvl9\" (UniqueName: \"kubernetes.io/projected/e8954104-beb7-4d35-96bb-f71f0df8e873-kube-api-access-zhvl9\") pod \"maas-api-revoke-sub-0d1a5c3d-7598d\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:47.132591 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:47.132559    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:32:47.198376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.198338    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:47.490067 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.490034    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"]
Oct 30 21:32:47.491869 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:47.491825    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8954104_beb7_4d35_96bb_f71f0df8e873.slice/crio-febcfabe5e0c6182f48dd6fef0e0611cdbc98ec0d19839fa04a955352202d351 WatchSource:0}: Error finding container febcfabe5e0c6182f48dd6fef0e0611cdbc98ec0d19839fa04a955352202d351: Status 404 returned error can't find the container with id febcfabe5e0c6182f48dd6fef0e0611cdbc98ec0d19839fa04a955352202d351
Oct 30 21:32:47.662399 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:47.662366    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abdc7e8-e62f-455f-94b4-603fb02a88f8" path="/var/lib/kubelet/pods/9abdc7e8-e62f-455f-94b4-603fb02a88f8/volumes"
Oct 30 21:32:48.320933 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:48.320897    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d" event={"ID":"e8954104-beb7-4d35-96bb-f71f0df8e873","Type":"ContainerStarted","Data":"26013cfbee9f64538ac24cb7526e88b44170c29dd23eaa2b71d7dfa1ea582379"}
Oct 30 21:32:48.321460 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:48.320951    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d" event={"ID":"e8954104-beb7-4d35-96bb-f71f0df8e873","Type":"ContainerStarted","Data":"febcfabe5e0c6182f48dd6fef0e0611cdbc98ec0d19839fa04a955352202d351"}
Oct 30 21:32:48.341003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:48.340951    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d" podStartSLOduration=2.340937818 podStartE2EDuration="2.340937818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:48.338046145 +0000 UTC m=+3334.829862692" watchObservedRunningTime="2026-10-30 21:32:48.340937818 +0000 UTC m=+3334.832754363"
Oct 30 21:32:48.701364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:48.701325    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"]
Oct 30 21:32:48.708920 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:48.708892    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6741b1-dgdfj"]
Oct 30 21:32:49.330418 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:49.330381    4810 generic.go:356] "Generic (PLEG): container finished" podID="e8954104-beb7-4d35-96bb-f71f0df8e873" containerID="26013cfbee9f64538ac24cb7526e88b44170c29dd23eaa2b71d7dfa1ea582379" exitCode=0
Oct 30 21:32:49.330902 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:49.330459    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d" event={"ID":"e8954104-beb7-4d35-96bb-f71f0df8e873","Type":"ContainerDied","Data":"26013cfbee9f64538ac24cb7526e88b44170c29dd23eaa2b71d7dfa1ea582379"}
Oct 30 21:32:49.648763 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:49.648717    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"]
Oct 30 21:32:49.652228 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:49.652205    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-bc102e-jhm2z"]
Oct 30 21:32:49.664507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:49.664478    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8906e24a-9968-4bcc-895e-6da393ac5ef4" path="/var/lib/kubelet/pods/8906e24a-9968-4bcc-895e-6da393ac5ef4/volumes"
Oct 30 21:32:49.664948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:49.664933    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfafe974-e665-48d1-b7e3-0d090e49f2fd" path="/var/lib/kubelet/pods/dfafe974-e665-48d1-b7e3-0d090e49f2fd/volumes"
Oct 30 21:32:50.490481 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:50.490452    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:50.558248 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:50.558213    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8954104-beb7-4d35-96bb-f71f0df8e873-kube-api-access-zhvl9\" (UniqueName: \"kubernetes.io/projected/e8954104-beb7-4d35-96bb-f71f0df8e873-kube-api-access-zhvl9\") pod \"e8954104-beb7-4d35-96bb-f71f0df8e873\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") "
Oct 30 21:32:50.558438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:50.558334    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-xks-ca\") pod \"e8954104-beb7-4d35-96bb-f71f0df8e873\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") "
Oct 30 21:32:50.558438 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:50.558415    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-openshift-ca\") pod \"e8954104-beb7-4d35-96bb-f71f0df8e873\" (UID: \"e8954104-beb7-4d35-96bb-f71f0df8e873\") "
Oct 30 21:32:50.558848 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:50.558816    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-openshift-ca" pod "e8954104-beb7-4d35-96bb-f71f0df8e873" (UID: "e8954104-beb7-4d35-96bb-f71f0df8e873"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:50.560577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:50.560545    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-xks-ca" pod "e8954104-beb7-4d35-96bb-f71f0df8e873" (UID: "e8954104-beb7-4d35-96bb-f71f0df8e873"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:50.560577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:50.560562    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8954104-beb7-4d35-96bb-f71f0df8e873-kube-api-access-zhvl9" pod "e8954104-beb7-4d35-96bb-f71f0df8e873" (UID: "e8954104-beb7-4d35-96bb-f71f0df8e873"). InnerVolumeSpecName "kube-api-access-zhvl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:50.659919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:50.659881    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:50.659919 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:50.659917    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8954104-beb7-4d35-96bb-f71f0df8e873-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:50.660148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:50.659931    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhvl9\" (UniqueName: \"kubernetes.io/projected/e8954104-beb7-4d35-96bb-f71f0df8e873-kube-api-access-zhvl9\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:51.347230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:51.347194    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d"
Oct 30 21:32:51.347230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:51.347215    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1a5c3d-7598d" event={"ID":"e8954104-beb7-4d35-96bb-f71f0df8e873","Type":"ContainerDied","Data":"febcfabe5e0c6182f48dd6fef0e0611cdbc98ec0d19839fa04a955352202d351"}
Oct 30 21:32:51.347503 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:51.347305    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="febcfabe5e0c6182f48dd6fef0e0611cdbc98ec0d19839fa04a955352202d351"
Oct 30 21:32:52.295823 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.295750    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:52.356008 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.355974    4810 generic.go:356] "Generic (PLEG): container finished" podID="176f280b-f3ff-4232-8984-639ab7fb711f" containerID="5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1" exitCode=0
Oct 30 21:32:52.356217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.356025    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"
Oct 30 21:32:52.356217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.356056    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" event={"ID":"176f280b-f3ff-4232-8984-639ab7fb711f","Type":"ContainerDied","Data":"5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1"}
Oct 30 21:32:52.356217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.356121    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6" event={"ID":"176f280b-f3ff-4232-8984-639ab7fb711f","Type":"ContainerDied","Data":"f863eeea10b88ed32a47ab25e6b0a0312c3ad33e6aa9ec0e094d8855a6bd1078"}
Oct 30 21:32:52.356217 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.356151    4810 scope.go:122] "RemoveContainer" containerID="5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1"
Oct 30 21:32:52.367157 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.367138    4810 scope.go:122] "RemoveContainer" containerID="5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1"
Oct 30 21:32:52.367460 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:52.367434    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1\": container with ID starting with 5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1 not found: ID does not exist" containerID="5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1"
Oct 30 21:32:52.367510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.367471    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1"} err="failed to get container status \"5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1\": rpc error: code = NotFound desc = could not find container \"5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1\": container with ID starting with 5e28be2d9cb6f65fe46bbd61f589427d5e87920ef2b24fbbf057862523b495a1 not found: ID does not exist"
Oct 30 21:32:52.381573 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.381551    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-certs\") pod \"176f280b-f3ff-4232-8984-639ab7fb711f\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") "
Oct 30 21:32:52.381700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.381604    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-socket\") pod \"176f280b-f3ff-4232-8984-639ab7fb711f\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") "
Oct 30 21:32:52.381700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.381649    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-istio-token\" (UniqueName: \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-istio-token\") pod \"176f280b-f3ff-4232-8984-639ab7fb711f\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") "
Oct 30 21:32:52.381700 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.381679    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-credential-socket\") pod \"176f280b-f3ff-4232-8984-639ab7fb711f\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") "
Oct 30 21:32:52.381896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.381784    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-data\") pod \"176f280b-f3ff-4232-8984-639ab7fb711f\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") "
Oct 30 21:32:52.381896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.381828    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-kube-api-access-8sv76\" (UniqueName: \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-kube-api-access-8sv76\") pod \"176f280b-f3ff-4232-8984-639ab7fb711f\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") "
Oct 30 21:32:52.382079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.381951    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-socket" pod "176f280b-f3ff-4232-8984-639ab7fb711f" (UID: "176f280b-f3ff-4232-8984-639ab7fb711f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:52.382079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.381971    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176f280b-f3ff-4232-8984-639ab7fb711f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176f280b-f3ff-4232-8984-639ab7fb711f-istiod-ca-cert\") pod \"176f280b-f3ff-4232-8984-639ab7fb711f\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") "
Oct 30 21:32:52.382079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382008    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-envoy\") pod \"176f280b-f3ff-4232-8984-639ab7fb711f\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") "
Oct 30 21:32:52.382079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382015    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-credential-socket" pod "176f280b-f3ff-4232-8984-639ab7fb711f" (UID: "176f280b-f3ff-4232-8984-639ab7fb711f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:52.382079 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382071    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/176f280b-f3ff-4232-8984-639ab7fb711f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176f280b-f3ff-4232-8984-639ab7fb711f-istio-podinfo\") pod \"176f280b-f3ff-4232-8984-639ab7fb711f\" (UID: \"176f280b-f3ff-4232-8984-639ab7fb711f\") "
Oct 30 21:32:52.382378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382136    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-data" pod "176f280b-f3ff-4232-8984-639ab7fb711f" (UID: "176f280b-f3ff-4232-8984-639ab7fb711f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:52.382378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382229    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-certs" pod "176f280b-f3ff-4232-8984-639ab7fb711f" (UID: "176f280b-f3ff-4232-8984-639ab7fb711f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:52.382378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382351    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176f280b-f3ff-4232-8984-639ab7fb711f-istiod-ca-cert" pod "176f280b-f3ff-4232-8984-639ab7fb711f" (UID: "176f280b-f3ff-4232-8984-639ab7fb711f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:52.382674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382631    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:52.382674 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382658    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:52.382816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382674    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:52.382816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382688    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:52.382816 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.382701    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/176f280b-f3ff-4232-8984-639ab7fb711f-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:52.384255 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.384234    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-envoy" pod "176f280b-f3ff-4232-8984-639ab7fb711f" (UID: "176f280b-f3ff-4232-8984-639ab7fb711f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:52.384367 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.384328    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/176f280b-f3ff-4232-8984-639ab7fb711f-istio-podinfo" pod "176f280b-f3ff-4232-8984-639ab7fb711f" (UID: "176f280b-f3ff-4232-8984-639ab7fb711f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:52.384447 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.384428    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-istio-token" pod "176f280b-f3ff-4232-8984-639ab7fb711f" (UID: "176f280b-f3ff-4232-8984-639ab7fb711f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:52.384502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.384450    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-kube-api-access-8sv76" pod "176f280b-f3ff-4232-8984-639ab7fb711f" (UID: "176f280b-f3ff-4232-8984-639ab7fb711f"). InnerVolumeSpecName "kube-api-access-8sv76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:52.483342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.483307    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/176f280b-f3ff-4232-8984-639ab7fb711f-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:52.483342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.483338    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/176f280b-f3ff-4232-8984-639ab7fb711f-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:52.483342 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.483348    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:52.483589 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.483357    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sv76\" (UniqueName: \"kubernetes.io/projected/176f280b-f3ff-4232-8984-639ab7fb711f-kube-api-access-8sv76\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:52.686309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.686280    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"]
Oct 30 21:32:52.691601 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:52.691579    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-84wg6"]
Oct 30 21:32:52.988579 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:52.988497    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:32:53.662009 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:53.661978    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" path="/var/lib/kubelet/pods/176f280b-f3ff-4232-8984-639ab7fb711f/volumes"
Oct 30 21:32:55.161047 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:32:55.161009    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:32:58.380761 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.380721    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"]
Oct 30 21:32:58.381490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.381459    4810 cpu_manager.go:414] "containerMap: removing container" podUID="e8954104-beb7-4d35-96bb-f71f0df8e873" containerName="revoke-keys"
Oct 30 21:32:58.381490 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.381480    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8954104-beb7-4d35-96bb-f71f0df8e873" containerName="revoke-keys"
Oct 30 21:32:58.381641 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.381526    4810 cpu_manager.go:414] "containerMap: removing container" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy"
Oct 30 21:32:58.381641 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.381536    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy"
Oct 30 21:32:58.381758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.381646    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8954104-beb7-4d35-96bb-f71f0df8e873" containerName="revoke-keys"
Oct 30 21:32:58.381758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.381661    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="176f280b-f3ff-4232-8984-639ab7fb711f" containerName="istio-proxy"
Oct 30 21:32:58.386508 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.386484    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:58.393944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.393919    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"]
Oct 30 21:32:58.456266 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.456234    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:58.456455 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.456291    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:58.456455 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.456380    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-444r8\" (UniqueName: \"kubernetes.io/projected/4c34c2c8-319b-4173-9e40-f9abce02dee1-kube-api-access-444r8\") pod \"maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:58.457030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.457006    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:58.457252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.457238    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\""
Oct 30 21:32:58.557386 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.557349    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:58.557563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.557395    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-444r8\" (UniqueName: \"kubernetes.io/projected/4c34c2c8-319b-4173-9e40-f9abce02dee1-kube-api-access-444r8\") pod \"maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:58.557563 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.557491    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:58.558167 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.558142    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:58.559956 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.559937    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:58.566382 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.566362    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-444r8\" (UniqueName: \"kubernetes.io/projected/4c34c2c8-319b-4173-9e40-f9abce02dee1-kube-api-access-444r8\") pod \"maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:58.638934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.638740    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"]
Oct 30 21:32:58.639120 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.639021    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy" containerID="cri-o://f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1" gracePeriod=30
Oct 30 21:32:58.713220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:58.713175    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:32:59.039937 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:59.039472    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"]
Oct 30 21:32:59.045152 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:32:59.045121    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c34c2c8_319b_4173_9e40_f9abce02dee1.slice/crio-bd586e74823571547cf4f6da636608fa0766060ff4fc186857da2b1a9d2c28f0 WatchSource:0}: Error finding container bd586e74823571547cf4f6da636608fa0766060ff4fc186857da2b1a9d2c28f0: Status 404 returned error can't find the container with id bd586e74823571547cf4f6da636608fa0766060ff4fc186857da2b1a9d2c28f0
Oct 30 21:32:59.429066 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:32:59.429033    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv" event={"ID":"4c34c2c8-319b-4173-9e40-f9abce02dee1","Type":"ContainerStarted","Data":"bd586e74823571547cf4f6da636608fa0766060ff4fc186857da2b1a9d2c28f0"}
Oct 30 21:33:00.439116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:00.439019    4810 generic.go:356] "Generic (PLEG): container finished" podID="4c34c2c8-319b-4173-9e40-f9abce02dee1" containerID="7647e49e1ed3730512521c0a0446584707dd96a82bcb1cdae7a727bcc250ae4a" exitCode=0
Oct 30 21:33:00.439116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:00.439069    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv" event={"ID":"4c34c2c8-319b-4173-9e40-f9abce02dee1","Type":"ContainerDied","Data":"7647e49e1ed3730512521c0a0446584707dd96a82bcb1cdae7a727bcc250ae4a"}
Oct 30 21:33:00.991710 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:00.991660    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"]
Oct 30 21:33:00.995013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:00.994988    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f9a819-p62zs"]
Oct 30 21:33:01.618090 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.618057    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:33:01.662382 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.662351    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9edee9a-22e5-4e20-85c4-be9d77dcd0bd" path="/var/lib/kubelet/pods/a9edee9a-22e5-4e20-85c4-be9d77dcd0bd/volumes"
Oct 30 21:33:01.696923 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.696888    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c34c2c8-319b-4173-9e40-f9abce02dee1-kube-api-access-444r8\" (UniqueName: \"kubernetes.io/projected/4c34c2c8-319b-4173-9e40-f9abce02dee1-kube-api-access-444r8\") pod \"4c34c2c8-319b-4173-9e40-f9abce02dee1\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") "
Oct 30 21:33:01.697099 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.697018    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-openshift-ca\") pod \"4c34c2c8-319b-4173-9e40-f9abce02dee1\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") "
Oct 30 21:33:01.697206 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.697192    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-xks-ca\") pod \"4c34c2c8-319b-4173-9e40-f9abce02dee1\" (UID: \"4c34c2c8-319b-4173-9e40-f9abce02dee1\") "
Oct 30 21:33:01.709758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.697669    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-openshift-ca" pod "4c34c2c8-319b-4173-9e40-f9abce02dee1" (UID: "4c34c2c8-319b-4173-9e40-f9abce02dee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:01.709758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.698235    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-openshift-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:01.709758 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.700344    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c34c2c8-319b-4173-9e40-f9abce02dee1-kube-api-access-444r8" pod "4c34c2c8-319b-4173-9e40-f9abce02dee1" (UID: "4c34c2c8-319b-4173-9e40-f9abce02dee1"). InnerVolumeSpecName "kube-api-access-444r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:01.711718 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.711571    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-xks-ca" pod "4c34c2c8-319b-4173-9e40-f9abce02dee1" (UID: "4c34c2c8-319b-4173-9e40-f9abce02dee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:33:01.799683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.799647    4810 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c34c2c8-319b-4173-9e40-f9abce02dee1-maas-api-xks-ca\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:01.799683 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:01.799680    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-444r8\" (UniqueName: \"kubernetes.io/projected/4c34c2c8-319b-4173-9e40-f9abce02dee1-kube-api-access-444r8\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:02.199668 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:33:02.199631    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:33:02.455934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:02.455814    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"
Oct 30 21:33:02.455934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:02.455843    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv" event={"ID":"4c34c2c8-319b-4173-9e40-f9abce02dee1","Type":"ContainerDied","Data":"bd586e74823571547cf4f6da636608fa0766060ff4fc186857da2b1a9d2c28f0"}
Oct 30 21:33:02.456125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:02.455969    4810 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd586e74823571547cf4f6da636608fa0766060ff4fc186857da2b1a9d2c28f0"
Oct 30 21:33:03.003777 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:33:03.003735    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:33:03.932955 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:03.931648    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 21:33:04.026148 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026108    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-kube-api-access-g5b4x\" (UniqueName: \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-kube-api-access-g5b4x\") pod \"bcf96d91-a658-426b-8ee0-e471f8ef6730\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") "
Oct 30 21:33:04.026627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026199    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf96d91-a658-426b-8ee0-e471f8ef6730-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf96d91-a658-426b-8ee0-e471f8ef6730-istiod-ca-cert\") pod \"bcf96d91-a658-426b-8ee0-e471f8ef6730\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") "
Oct 30 21:33:04.026627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026319    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-data\") pod \"bcf96d91-a658-426b-8ee0-e471f8ef6730\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") "
Oct 30 21:33:04.026627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026469    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-credential-socket\") pod \"bcf96d91-a658-426b-8ee0-e471f8ef6730\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") "
Oct 30 21:33:04.026627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026516    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-certs\") pod \"bcf96d91-a658-426b-8ee0-e471f8ef6730\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") "
Oct 30 21:33:04.026627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026542    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-data" pod "bcf96d91-a658-426b-8ee0-e471f8ef6730" (UID: "bcf96d91-a658-426b-8ee0-e471f8ef6730"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:04.026627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026573    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-podinfo\") pod \"bcf96d91-a658-426b-8ee0-e471f8ef6730\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") "
Oct 30 21:33:04.026627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026602    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-socket\") pod \"bcf96d91-a658-426b-8ee0-e471f8ef6730\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") "
Oct 30 21:33:04.026627 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026613    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf96d91-a658-426b-8ee0-e471f8ef6730-istiod-ca-cert" pod "bcf96d91-a658-426b-8ee0-e471f8ef6730" (UID: "bcf96d91-a658-426b-8ee0-e471f8ef6730"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:04.027069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026631    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-token\" (UniqueName: \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-token\") pod \"bcf96d91-a658-426b-8ee0-e471f8ef6730\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") "
Oct 30 21:33:04.027069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026685    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-envoy\") pod \"bcf96d91-a658-426b-8ee0-e471f8ef6730\" (UID: \"bcf96d91-a658-426b-8ee0-e471f8ef6730\") "
Oct 30 21:33:04.027069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026760    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-certs" pod "bcf96d91-a658-426b-8ee0-e471f8ef6730" (UID: "bcf96d91-a658-426b-8ee0-e471f8ef6730"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:04.027069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026776    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-credential-socket" pod "bcf96d91-a658-426b-8ee0-e471f8ef6730" (UID: "bcf96d91-a658-426b-8ee0-e471f8ef6730"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:04.027069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.026926    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-socket" pod "bcf96d91-a658-426b-8ee0-e471f8ef6730" (UID: "bcf96d91-a658-426b-8ee0-e471f8ef6730"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:04.027362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.027154    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf96d91-a658-426b-8ee0-e471f8ef6730-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:04.027362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.027180    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:04.027362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.027195    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:04.027362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.027209    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:04.027362 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.027222    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:04.028750 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.028719    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-podinfo" pod "bcf96d91-a658-426b-8ee0-e471f8ef6730" (UID: "bcf96d91-a658-426b-8ee0-e471f8ef6730"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:33:04.028926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.028853    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-kube-api-access-g5b4x" pod "bcf96d91-a658-426b-8ee0-e471f8ef6730" (UID: "bcf96d91-a658-426b-8ee0-e471f8ef6730"). InnerVolumeSpecName "kube-api-access-g5b4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:04.029001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.028934    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-envoy" pod "bcf96d91-a658-426b-8ee0-e471f8ef6730" (UID: "bcf96d91-a658-426b-8ee0-e471f8ef6730"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:04.029001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.028986    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-token" pod "bcf96d91-a658-426b-8ee0-e471f8ef6730" (UID: "bcf96d91-a658-426b-8ee0-e471f8ef6730"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:04.128249 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.128213    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:04.128249 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.128242    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:04.128249 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.128255    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf96d91-a658-426b-8ee0-e471f8ef6730-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:04.128510 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.128268    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5b4x\" (UniqueName: \"kubernetes.io/projected/bcf96d91-a658-426b-8ee0-e471f8ef6730-kube-api-access-g5b4x\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:04.472749 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.472710    4810 generic.go:356] "Generic (PLEG): container finished" podID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerID="f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1" exitCode=0
Oct 30 21:33:04.472997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.472751    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" event={"ID":"bcf96d91-a658-426b-8ee0-e471f8ef6730","Type":"ContainerDied","Data":"f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1"}
Oct 30 21:33:04.472997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.472771    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"
Oct 30 21:33:04.472997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.472822    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t" event={"ID":"bcf96d91-a658-426b-8ee0-e471f8ef6730","Type":"ContainerDied","Data":"db79764c459fb7c6c83f3855818905614ba682e9ccef705499a55907da60e984"}
Oct 30 21:33:04.472997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.472853    4810 scope.go:122] "RemoveContainer" containerID="f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1"
Oct 30 21:33:04.485507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.485485    4810 scope.go:122] "RemoveContainer" containerID="f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1"
Oct 30 21:33:04.485890 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:33:04.485841    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1\": container with ID starting with f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1 not found: ID does not exist" containerID="f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1"
Oct 30 21:33:04.485997 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.485898    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1"} err="failed to get container status \"f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1\": rpc error: code = NotFound desc = could not find container \"f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1\": container with ID starting with f025df37f72298def7a346ef155909fc565272c4209738698721108595461ac1 not found: ID does not exist"
Oct 30 21:33:04.498590 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.498565    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"]
Oct 30 21:33:04.502389 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:04.502365    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-dkw9t"]
Oct 30 21:33:05.661279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:05.661248    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" path="/var/lib/kubelet/pods/bcf96d91-a658-426b-8ee0-e471f8ef6730/volumes"
Oct 30 21:33:06.816576 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:06.816537    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:33:08.503279 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:08.503238    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"]
Oct 30 21:33:08.514948 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:08.514912    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"]
Oct 30 21:33:10.062361 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:10.062328    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"]
Oct 30 21:33:10.066711 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:10.066678    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d95261-4tphv"]
Oct 30 21:33:10.565977 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:10.565925    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"]
Oct 30 21:33:10.569355 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:10.569139    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerName="istio-proxy" containerID="cri-o://0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856" gracePeriod=30
Oct 30 21:33:11.661971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:11.661937    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c34c2c8-319b-4173-9e40-f9abce02dee1" path="/var/lib/kubelet/pods/4c34c2c8-319b-4173-9e40-f9abce02dee1/volumes"
Oct 30 21:33:13.082402 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:33:13.082363    4810 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a273_e5af_4bfc_b365_0d3fb804e5c4.slice/crio-8beb4250fee78c634a9d0a5c2f5c5484755901816f34c43386cb8d7c342e26fe.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:33:13.504401 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:13.504359    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk" podUID="1c20507b-2dd0-4490-b5eb-dc475f769a04" containerName="payload-processing" containerID="cri-o://db080dffaadef9ac3cb966393d7ea15490a4a43a8f039f3fbc1e22bc952f3bbf" gracePeriod=25
Oct 30 21:33:13.515953 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:13.515920    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z" podUID="297f5eb4-df4e-47dc-bb9b-41505d5da2be" containerName="payload-pre-processing" containerID="cri-o://abfe8b516bbba1aa81460876caede98edefbb6690a5366e03e4cb35aaa80f370" gracePeriod=25
Oct 30 21:33:15.780368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.780337    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:33:15.874692 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.874658    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 21:33:15.975514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.975468    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istiod-ca-cert\") pod \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") "
Oct 30 21:33:15.975715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.975524    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-credential-socket\") pod \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") "
Oct 30 21:33:15.975715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.975624    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-token\" (UniqueName: \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-token\") pod \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") "
Oct 30 21:33:15.975715 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.975685    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-envoy\") pod \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") "
Oct 30 21:33:15.975932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.975740    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-data\") pod \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") "
Oct 30 21:33:15.975932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.975845    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-kube-api-access-wnmrj\" (UniqueName: \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-kube-api-access-wnmrj\") pod \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") "
Oct 30 21:33:15.975932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.975891    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-credential-socket" pod "f4234e57-d237-4b81-b8b6-a1ea7e75e12a" (UID: "f4234e57-d237-4b81-b8b6-a1ea7e75e12a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:15.975932 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.975902    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-podinfo\") pod \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") "
Oct 30 21:33:15.976195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.975944    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istiod-ca-cert" pod "f4234e57-d237-4b81-b8b6-a1ea7e75e12a" (UID: "f4234e57-d237-4b81-b8b6-a1ea7e75e12a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:15.976195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.975984    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-certs\") pod \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") "
Oct 30 21:33:15.976195 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.976030    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-socket\") pod \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\" (UID: \"f4234e57-d237-4b81-b8b6-a1ea7e75e12a\") "
Oct 30 21:33:15.976397 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.976374    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-certs" pod "f4234e57-d237-4b81-b8b6-a1ea7e75e12a" (UID: "f4234e57-d237-4b81-b8b6-a1ea7e75e12a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:15.976502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.976461    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-data" pod "f4234e57-d237-4b81-b8b6-a1ea7e75e12a" (UID: "f4234e57-d237-4b81-b8b6-a1ea7e75e12a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:15.976502 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.976492    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-socket" pod "f4234e57-d237-4b81-b8b6-a1ea7e75e12a" (UID: "f4234e57-d237-4b81-b8b6-a1ea7e75e12a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:15.976666 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.976650    4810 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-data\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:15.976709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.976668    4810 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:15.976709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.976678    4810 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-workload-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:15.976709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.976687    4810 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istiod-ca-cert\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:15.976709 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.976697    4810 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-credential-socket\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:15.978600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.978578    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-token" pod "f4234e57-d237-4b81-b8b6-a1ea7e75e12a" (UID: "f4234e57-d237-4b81-b8b6-a1ea7e75e12a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:15.978752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.978614    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-envoy" pod "f4234e57-d237-4b81-b8b6-a1ea7e75e12a" (UID: "f4234e57-d237-4b81-b8b6-a1ea7e75e12a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:15.978752 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.978658    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-podinfo" pod "f4234e57-d237-4b81-b8b6-a1ea7e75e12a" (UID: "f4234e57-d237-4b81-b8b6-a1ea7e75e12a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:33:15.980309 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:15.980283    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-kube-api-access-wnmrj" pod "f4234e57-d237-4b81-b8b6-a1ea7e75e12a" (UID: "f4234e57-d237-4b81-b8b6-a1ea7e75e12a"). InnerVolumeSpecName "kube-api-access-wnmrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:16.078093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.078053    4810 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-token\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:16.078093 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.078094    4810 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-envoy\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:16.078378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.078111    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnmrj\" (UniqueName: \"kubernetes.io/projected/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-kube-api-access-wnmrj\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:16.078378 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.078126    4810 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4234e57-d237-4b81-b8b6-a1ea7e75e12a-istio-podinfo\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:16.595579 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.595539    4810 generic.go:356] "Generic (PLEG): container finished" podID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerID="0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856" exitCode=0
Oct 30 21:33:16.595773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.595590    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"
Oct 30 21:33:16.595773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.595622    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" event={"ID":"f4234e57-d237-4b81-b8b6-a1ea7e75e12a","Type":"ContainerDied","Data":"0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856"}
Oct 30 21:33:16.595773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.595682    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc" event={"ID":"f4234e57-d237-4b81-b8b6-a1ea7e75e12a","Type":"ContainerDied","Data":"44e4882e6155101e7391258335fbad8918b1f0066fae0c3865ba9caee615c986"}
Oct 30 21:33:16.595773 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.595714    4810 scope.go:122] "RemoveContainer" containerID="0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856"
Oct 30 21:33:16.615507 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.615401    4810 scope.go:122] "RemoveContainer" containerID="0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856"
Oct 30 21:33:16.615781 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:33:16.615730    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856\": container with ID starting with 0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856 not found: ID does not exist" containerID="0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856"
Oct 30 21:33:16.615903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.615772    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856"} err="failed to get container status \"0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856\": rpc error: code = NotFound desc = could not find container \"0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856\": container with ID starting with 0f5275362a5537435ab3e34f3e11867bfcd5379e6e9f579e3c628520dedec856 not found: ID does not exist"
Oct 30 21:33:16.634139 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.634114    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"]
Oct 30 21:33:16.639224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:16.639196    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-5rfrc"]
Oct 30 21:33:17.662962 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:17.662925    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" path="/var/lib/kubelet/pods/f4234e57-d237-4b81-b8b6-a1ea7e75e12a/volumes"
Oct 30 21:33:19.756009 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:19.755977    4810 scope.go:122] "RemoveContainer" containerID="60f78cec43ffbccaf91c4344d58bcb62e5b35fd1f8dfd1e47617e57e31222280"
Oct 30 21:33:19.766026 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:19.766006    4810 scope.go:122] "RemoveContainer" containerID="7d53807d4ee7e0030bd10bef5b00b7a3e323d365787c3235bc0788c2462553d6"
Oct 30 21:33:20.256426 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:20.256388    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z" podUID="297f5eb4-df4e-47dc-bb9b-41505d5da2be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:20.256710 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:20.256400    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk" podUID="1c20507b-2dd0-4490-b5eb-dc475f769a04" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:25.156486 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:33:25.156402    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:33:29.174972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:29.174930    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"]
Oct 30 21:33:29.193109 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:29.193070    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"]
Oct 30 21:33:30.255824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:30.255781    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z" podUID="297f5eb4-df4e-47dc-bb9b-41505d5da2be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:30.255824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:30.255809    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk" podUID="1c20507b-2dd0-4490-b5eb-dc475f769a04" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:33.733125 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.733089    4810 generic.go:356] "Generic (PLEG): container finished" podID="1c20507b-2dd0-4490-b5eb-dc475f769a04" containerID="db080dffaadef9ac3cb966393d7ea15490a4a43a8f039f3fbc1e22bc952f3bbf" exitCode=0
Oct 30 21:33:33.733529 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.733163    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk" event={"ID":"1c20507b-2dd0-4490-b5eb-dc475f769a04","Type":"ContainerDied","Data":"db080dffaadef9ac3cb966393d7ea15490a4a43a8f039f3fbc1e22bc952f3bbf"}
Oct 30 21:33:33.735670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.735440    4810 generic.go:356] "Generic (PLEG): container finished" podID="297f5eb4-df4e-47dc-bb9b-41505d5da2be" containerID="abfe8b516bbba1aa81460876caede98edefbb6690a5366e03e4cb35aaa80f370" exitCode=0
Oct 30 21:33:33.735670 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.735624    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z" event={"ID":"297f5eb4-df4e-47dc-bb9b-41505d5da2be","Type":"ContainerDied","Data":"abfe8b516bbba1aa81460876caede98edefbb6690a5366e03e4cb35aaa80f370"}
Oct 30 21:33:33.797203 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.797173    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:33:33.831059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.831032    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:33:33.859445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.859410    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297f5eb4-df4e-47dc-bb9b-41505d5da2be-kube-api-access-z5bwh\" (UniqueName: \"kubernetes.io/projected/297f5eb4-df4e-47dc-bb9b-41505d5da2be-kube-api-access-z5bwh\") pod \"297f5eb4-df4e-47dc-bb9b-41505d5da2be\" (UID: \"297f5eb4-df4e-47dc-bb9b-41505d5da2be\") "
Oct 30 21:33:33.859608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.859556    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c20507b-2dd0-4490-b5eb-dc475f769a04-kube-api-access-vw7x5\" (UniqueName: \"kubernetes.io/projected/1c20507b-2dd0-4490-b5eb-dc475f769a04-kube-api-access-vw7x5\") pod \"1c20507b-2dd0-4490-b5eb-dc475f769a04\" (UID: \"1c20507b-2dd0-4490-b5eb-dc475f769a04\") "
Oct 30 21:33:33.859682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.859606    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c20507b-2dd0-4490-b5eb-dc475f769a04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c20507b-2dd0-4490-b5eb-dc475f769a04-plugins-config-volume\") pod \"1c20507b-2dd0-4490-b5eb-dc475f769a04\" (UID: \"1c20507b-2dd0-4490-b5eb-dc475f769a04\") "
Oct 30 21:33:33.859682 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.859638    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297f5eb4-df4e-47dc-bb9b-41505d5da2be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/297f5eb4-df4e-47dc-bb9b-41505d5da2be-plugins-config-volume\") pod \"297f5eb4-df4e-47dc-bb9b-41505d5da2be\" (UID: \"297f5eb4-df4e-47dc-bb9b-41505d5da2be\") "
Oct 30 21:33:33.860116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.860089    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297f5eb4-df4e-47dc-bb9b-41505d5da2be-plugins-config-volume" pod "297f5eb4-df4e-47dc-bb9b-41505d5da2be" (UID: "297f5eb4-df4e-47dc-bb9b-41505d5da2be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:33.860116 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.860092    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c20507b-2dd0-4490-b5eb-dc475f769a04-plugins-config-volume" pod "1c20507b-2dd0-4490-b5eb-dc475f769a04" (UID: "1c20507b-2dd0-4490-b5eb-dc475f769a04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:33.861629 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.861603    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297f5eb4-df4e-47dc-bb9b-41505d5da2be-kube-api-access-z5bwh" pod "297f5eb4-df4e-47dc-bb9b-41505d5da2be" (UID: "297f5eb4-df4e-47dc-bb9b-41505d5da2be"). InnerVolumeSpecName "kube-api-access-z5bwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:33.862189 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.862164    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c20507b-2dd0-4490-b5eb-dc475f769a04-kube-api-access-vw7x5" pod "1c20507b-2dd0-4490-b5eb-dc475f769a04" (UID: "1c20507b-2dd0-4490-b5eb-dc475f769a04"). InnerVolumeSpecName "kube-api-access-vw7x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:33.960873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.960829    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5bwh\" (UniqueName: \"kubernetes.io/projected/297f5eb4-df4e-47dc-bb9b-41505d5da2be-kube-api-access-z5bwh\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:33.961054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.960892    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7x5\" (UniqueName: \"kubernetes.io/projected/1c20507b-2dd0-4490-b5eb-dc475f769a04-kube-api-access-vw7x5\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:33.961054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.960903    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c20507b-2dd0-4490-b5eb-dc475f769a04-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:33.961054 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:33.960912    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/297f5eb4-df4e-47dc-bb9b-41505d5da2be-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:34.175612 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.175562    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4" podUID="212bd647-b861-4804-88a0-a3cd4372ece2" containerName="payload-processing" containerID="cri-o://de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8" gracePeriod=25
Oct 30 21:33:34.194000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.193960    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst" podUID="0e74b3bb-da56-4fe7-9585-916338363efb" containerName="payload-pre-processing" containerID="cri-o://5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3" gracePeriod=25
Oct 30 21:33:34.744824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.744789    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"
Oct 30 21:33:34.744824 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.744805    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk" event={"ID":"1c20507b-2dd0-4490-b5eb-dc475f769a04","Type":"ContainerDied","Data":"3e03d5f6ef947cacb64e0f296786bdfb042d1e38113f37ca4edb1ef61cecbde3"}
Oct 30 21:33:34.745369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.744897    4810 scope.go:122] "RemoveContainer" containerID="db080dffaadef9ac3cb966393d7ea15490a4a43a8f039f3fbc1e22bc952f3bbf"
Oct 30 21:33:34.746516 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.746498    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"
Oct 30 21:33:34.746634 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.746529    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z" event={"ID":"297f5eb4-df4e-47dc-bb9b-41505d5da2be","Type":"ContainerDied","Data":"09356e5fd150c45113e9c7ef3515c9856dcc8858f356294479a19dff1ade4189"}
Oct 30 21:33:34.757388 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.757368    4810 scope.go:122] "RemoveContainer" containerID="abfe8b516bbba1aa81460876caede98edefbb6690a5366e03e4cb35aaa80f370"
Oct 30 21:33:34.770732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.770711    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"]
Oct 30 21:33:34.773738 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.773685    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:33:34.776262 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.776239    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d95261-69665dc795-nz2vk"]
Oct 30 21:33:34.790420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.790394    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"]
Oct 30 21:33:34.792938 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:34.792913    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d95261-7f85f5cf87-bc65z"]
Oct 30 21:33:35.663605 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:35.663568    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c20507b-2dd0-4490-b5eb-dc475f769a04" path="/var/lib/kubelet/pods/1c20507b-2dd0-4490-b5eb-dc475f769a04/volumes"
Oct 30 21:33:35.664043 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:35.664029    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297f5eb4-df4e-47dc-bb9b-41505d5da2be" path="/var/lib/kubelet/pods/297f5eb4-df4e-47dc-bb9b-41505d5da2be/volumes"
Oct 30 21:33:39.414097 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:39.414055    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4" podUID="212bd647-b861-4804-88a0-a3cd4372ece2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:39.414632 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:39.414060    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst" podUID="0e74b3bb-da56-4fe7-9585-916338363efb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:49.414608 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:49.414564    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst" podUID="0e74b3bb-da56-4fe7-9585-916338363efb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:49.415072 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:49.414564    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4" podUID="212bd647-b861-4804-88a0-a3cd4372ece2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:50.565074 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:50.565030    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"]
Oct 30 21:33:50.579756 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:50.579715    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"]
Oct 30 21:33:54.469354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.469326    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:33:54.495630 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.494776    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:33:54.576806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.576776    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212bd647-b861-4804-88a0-a3cd4372ece2-kube-api-access-xktcs\" (UniqueName: \"kubernetes.io/projected/212bd647-b861-4804-88a0-a3cd4372ece2-kube-api-access-xktcs\") pod \"212bd647-b861-4804-88a0-a3cd4372ece2\" (UID: \"212bd647-b861-4804-88a0-a3cd4372ece2\") "
Oct 30 21:33:54.577027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.576932    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212bd647-b861-4804-88a0-a3cd4372ece2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212bd647-b861-4804-88a0-a3cd4372ece2-plugins-config-volume\") pod \"212bd647-b861-4804-88a0-a3cd4372ece2\" (UID: \"212bd647-b861-4804-88a0-a3cd4372ece2\") "
Oct 30 21:33:54.577415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.577384    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212bd647-b861-4804-88a0-a3cd4372ece2-plugins-config-volume" pod "212bd647-b861-4804-88a0-a3cd4372ece2" (UID: "212bd647-b861-4804-88a0-a3cd4372ece2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:54.579003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.578970    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212bd647-b861-4804-88a0-a3cd4372ece2-kube-api-access-xktcs" pod "212bd647-b861-4804-88a0-a3cd4372ece2" (UID: "212bd647-b861-4804-88a0-a3cd4372ece2"). InnerVolumeSpecName "kube-api-access-xktcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:54.659690 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.659656    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:33:54.678415 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.678382    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e74b3bb-da56-4fe7-9585-916338363efb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e74b3bb-da56-4fe7-9585-916338363efb-plugins-config-volume\") pod \"0e74b3bb-da56-4fe7-9585-916338363efb\" (UID: \"0e74b3bb-da56-4fe7-9585-916338363efb\") "
Oct 30 21:33:54.678587 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.678523    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e74b3bb-da56-4fe7-9585-916338363efb-kube-api-access-vgkxf\" (UniqueName: \"kubernetes.io/projected/0e74b3bb-da56-4fe7-9585-916338363efb-kube-api-access-vgkxf\") pod \"0e74b3bb-da56-4fe7-9585-916338363efb\" (UID: \"0e74b3bb-da56-4fe7-9585-916338363efb\") "
Oct 30 21:33:54.678780 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.678754    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e74b3bb-da56-4fe7-9585-916338363efb-plugins-config-volume" pod "0e74b3bb-da56-4fe7-9585-916338363efb" (UID: "0e74b3bb-da56-4fe7-9585-916338363efb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:54.678887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.678768    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xktcs\" (UniqueName: \"kubernetes.io/projected/212bd647-b861-4804-88a0-a3cd4372ece2-kube-api-access-xktcs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:54.678887 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.678812    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212bd647-b861-4804-88a0-a3cd4372ece2-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:54.680770 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.680741    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e74b3bb-da56-4fe7-9585-916338363efb-kube-api-access-vgkxf" pod "0e74b3bb-da56-4fe7-9585-916338363efb" (UID: "0e74b3bb-da56-4fe7-9585-916338363efb"). InnerVolumeSpecName "kube-api-access-vgkxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:54.780437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.780329    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgkxf\" (UniqueName: \"kubernetes.io/projected/0e74b3bb-da56-4fe7-9585-916338363efb-kube-api-access-vgkxf\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:54.780437 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.780358    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e74b3bb-da56-4fe7-9585-916338363efb-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:54.891327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.891291    4810 generic.go:356] "Generic (PLEG): container finished" podID="212bd647-b861-4804-88a0-a3cd4372ece2" containerID="de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8" exitCode=0
Oct 30 21:33:54.891492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.891336    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"
Oct 30 21:33:54.891492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.891381    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4" event={"ID":"212bd647-b861-4804-88a0-a3cd4372ece2","Type":"ContainerDied","Data":"de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8"}
Oct 30 21:33:54.891492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.891449    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4" event={"ID":"212bd647-b861-4804-88a0-a3cd4372ece2","Type":"ContainerDied","Data":"8098e9da60e9153b7c47721da3c01ebb3e6b52f98fb9dd60764a55068cb9ba56"}
Oct 30 21:33:54.891492 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.891483    4810 scope.go:122] "RemoveContainer" containerID="de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8"
Oct 30 21:33:54.893245 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.893224    4810 generic.go:356] "Generic (PLEG): container finished" podID="0e74b3bb-da56-4fe7-9585-916338363efb" containerID="5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3" exitCode=0
Oct 30 21:33:54.893337 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.893276    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"
Oct 30 21:33:54.893403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.893283    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst" event={"ID":"0e74b3bb-da56-4fe7-9585-916338363efb","Type":"ContainerDied","Data":"5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3"}
Oct 30 21:33:54.893403 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.893396    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst" event={"ID":"0e74b3bb-da56-4fe7-9585-916338363efb","Type":"ContainerDied","Data":"8d7e13011d8430884810eb572392d0949ac93f4a1f618364a990267a3888e22f"}
Oct 30 21:33:54.907408 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.907389    4810 scope.go:122] "RemoveContainer" containerID="de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8"
Oct 30 21:33:54.907766 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:33:54.907737    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8\": container with ID starting with de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8 not found: ID does not exist" containerID="de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8"
Oct 30 21:33:54.908680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.907777    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8"} err="failed to get container status \"de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8\": rpc error: code = NotFound desc = could not find container \"de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8\": container with ID starting with de2c0bc882fbb2dde7d69484df1e3a7f3be570af2b533f8160de2577beb992f8 not found: ID does not exist"
Oct 30 21:33:54.908680 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.907799    4810 scope.go:122] "RemoveContainer" containerID="5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3"
Oct 30 21:33:54.917836 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.917812    4810 scope.go:122] "RemoveContainer" containerID="5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3"
Oct 30 21:33:54.918223 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:33:54.918202    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3\": container with ID starting with 5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3 not found: ID does not exist" containerID="5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3"
Oct 30 21:33:54.918314 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.918233    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3"} err="failed to get container status \"5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3\": rpc error: code = NotFound desc = could not find container \"5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3\": container with ID starting with 5498a7709ce550e30ec2ec2a6779bc9223c36935ba4585c61b6746de011773e3 not found: ID does not exist"
Oct 30 21:33:54.921012 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.920988    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"]
Oct 30 21:33:54.925547 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.925525    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e53c37-7c4c57b895-cftg4"]
Oct 30 21:33:54.937950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.937923    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"]
Oct 30 21:33:54.944672 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:54.944650    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e53c37-69cf6c7887-bwjst"]
Oct 30 21:33:55.155343 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:33:55.155308    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:33:55.566269 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:55.566173    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp" podUID="224bb67a-9e62-48c5-b703-a8de47cf7f92" containerName="payload-processing" containerID="cri-o://5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18" gracePeriod=25
Oct 30 21:33:55.580344 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:55.580287    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g" podUID="cd085a4e-4d27-4397-b0db-7f010b6b4cad" containerName="payload-pre-processing" containerID="cri-o://c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020" gracePeriod=25
Oct 30 21:33:55.662545 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:55.662504    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e74b3bb-da56-4fe7-9585-916338363efb" path="/var/lib/kubelet/pods/0e74b3bb-da56-4fe7-9585-916338363efb/volumes"
Oct 30 21:33:55.662934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:55.662917    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212bd647-b861-4804-88a0-a3cd4372ece2" path="/var/lib/kubelet/pods/212bd647-b861-4804-88a0-a3cd4372ece2/volumes"
Oct 30 21:33:56.171479 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:56.171437    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp" podUID="224bb67a-9e62-48c5-b703-a8de47cf7f92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:33:56.171652 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:33:56.171488    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g" podUID="cd085a4e-4d27-4397-b0db-7f010b6b4cad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:34:05.118702 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:05.118616    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"]
Oct 30 21:34:05.124621 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:05.124588    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"]
Oct 30 21:34:05.144884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:05.144837    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"]
Oct 30 21:34:06.170899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:06.170840    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g" podUID="cd085a4e-4d27-4397-b0db-7f010b6b4cad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:34:06.171299 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:06.170931    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp" podUID="224bb67a-9e62-48c5-b703-a8de47cf7f92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:34:15.858934 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.858904    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:34:15.893290 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.893256    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/224bb67a-9e62-48c5-b703-a8de47cf7f92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224bb67a-9e62-48c5-b703-a8de47cf7f92-plugins-config-volume\") pod \"224bb67a-9e62-48c5-b703-a8de47cf7f92\" (UID: \"224bb67a-9e62-48c5-b703-a8de47cf7f92\") "
Oct 30 21:34:15.893445 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.893416    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224bb67a-9e62-48c5-b703-a8de47cf7f92-kube-api-access-5qnsx\" (UniqueName: \"kubernetes.io/projected/224bb67a-9e62-48c5-b703-a8de47cf7f92-kube-api-access-5qnsx\") pod \"224bb67a-9e62-48c5-b703-a8de47cf7f92\" (UID: \"224bb67a-9e62-48c5-b703-a8de47cf7f92\") "
Oct 30 21:34:15.893993 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.893825    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224bb67a-9e62-48c5-b703-a8de47cf7f92-plugins-config-volume" pod "224bb67a-9e62-48c5-b703-a8de47cf7f92" (UID: "224bb67a-9e62-48c5-b703-a8de47cf7f92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:15.896356 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.896330    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224bb67a-9e62-48c5-b703-a8de47cf7f92-kube-api-access-5qnsx" pod "224bb67a-9e62-48c5-b703-a8de47cf7f92" (UID: "224bb67a-9e62-48c5-b703-a8de47cf7f92"). InnerVolumeSpecName "kube-api-access-5qnsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:15.902427 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.902404    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:34:15.994466 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.994435    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd085a4e-4d27-4397-b0db-7f010b6b4cad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd085a4e-4d27-4397-b0db-7f010b6b4cad-plugins-config-volume\") pod \"cd085a4e-4d27-4397-b0db-7f010b6b4cad\" (UID: \"cd085a4e-4d27-4397-b0db-7f010b6b4cad\") "
Oct 30 21:34:15.994661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.994582    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd085a4e-4d27-4397-b0db-7f010b6b4cad-kube-api-access-77v79\" (UniqueName: \"kubernetes.io/projected/cd085a4e-4d27-4397-b0db-7f010b6b4cad-kube-api-access-77v79\") pod \"cd085a4e-4d27-4397-b0db-7f010b6b4cad\" (UID: \"cd085a4e-4d27-4397-b0db-7f010b6b4cad\") "
Oct 30 21:34:15.994927 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.994897    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd085a4e-4d27-4397-b0db-7f010b6b4cad-plugins-config-volume" pod "cd085a4e-4d27-4397-b0db-7f010b6b4cad" (UID: "cd085a4e-4d27-4397-b0db-7f010b6b4cad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:15.995104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.995058    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnsx\" (UniqueName: \"kubernetes.io/projected/224bb67a-9e62-48c5-b703-a8de47cf7f92-kube-api-access-5qnsx\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:15.995104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.995080    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224bb67a-9e62-48c5-b703-a8de47cf7f92-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:15.995104 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.995094    4810 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd085a4e-4d27-4397-b0db-7f010b6b4cad-plugins-config-volume\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:15.997463 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:15.997437    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd085a4e-4d27-4397-b0db-7f010b6b4cad-kube-api-access-77v79" pod "cd085a4e-4d27-4397-b0db-7f010b6b4cad" (UID: "cd085a4e-4d27-4397-b0db-7f010b6b4cad"). InnerVolumeSpecName "kube-api-access-77v79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:16.060364 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.060327    4810 generic.go:356] "Generic (PLEG): container finished" podID="224bb67a-9e62-48c5-b703-a8de47cf7f92" containerID="5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18" exitCode=0
Oct 30 21:34:16.060543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.060377    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"
Oct 30 21:34:16.060543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.060420    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp" event={"ID":"224bb67a-9e62-48c5-b703-a8de47cf7f92","Type":"ContainerDied","Data":"5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18"}
Oct 30 21:34:16.060543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.060470    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp" event={"ID":"224bb67a-9e62-48c5-b703-a8de47cf7f92","Type":"ContainerDied","Data":"f12e48f198417034a8b068f9c170b33915adb0cfb746430e219456c2243dd6b3"}
Oct 30 21:34:16.060543 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.060491    4810 scope.go:122] "RemoveContainer" containerID="5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18"
Oct 30 21:34:16.061786 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.061763    4810 generic.go:356] "Generic (PLEG): container finished" podID="cd085a4e-4d27-4397-b0db-7f010b6b4cad" containerID="c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020" exitCode=0
Oct 30 21:34:16.061896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.061809    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"
Oct 30 21:34:16.061896 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.061868    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g" event={"ID":"cd085a4e-4d27-4397-b0db-7f010b6b4cad","Type":"ContainerDied","Data":"c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020"}
Oct 30 21:34:16.061999 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.061918    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g" event={"ID":"cd085a4e-4d27-4397-b0db-7f010b6b4cad","Type":"ContainerDied","Data":"79f6b4be7b35a70f3c5523f0269f529831b77d3054671e57b85cd2eea45acd28"}
Oct 30 21:34:16.071522 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.071499    4810 scope.go:122] "RemoveContainer" containerID="5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18"
Oct 30 21:34:16.071921 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:34:16.071899    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18\": container with ID starting with 5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18 not found: ID does not exist" containerID="5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18"
Oct 30 21:34:16.072023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.071928    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18"} err="failed to get container status \"5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18\": rpc error: code = NotFound desc = could not find container \"5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18\": container with ID starting with 5fbb64eadc50a4e57e3ffeb9a7b58eb55c5ff4e2ed143c30c0ff1cba40a63c18 not found: ID does not exist"
Oct 30 21:34:16.072023 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.071947    4810 scope.go:122] "RemoveContainer" containerID="c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020"
Oct 30 21:34:16.083723 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.083700    4810 scope.go:122] "RemoveContainer" containerID="c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020"
Oct 30 21:34:16.084063 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:34:16.084032    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020\": container with ID starting with c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020 not found: ID does not exist" containerID="c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020"
Oct 30 21:34:16.084163 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.084073    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020"} err="failed to get container status \"c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020\": rpc error: code = NotFound desc = could not find container \"c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020\": container with ID starting with c57c364c8ffef248d73cafab16b5bf7dd34794ab8c281a21f0041f187aaf0020 not found: ID does not exist"
Oct 30 21:34:16.087734 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.087708    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"]
Oct 30 21:34:16.094345 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.094323    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-321ccd-55fb65b87c-8w6rp"]
Oct 30 21:34:16.095903 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.095848    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77v79\" (UniqueName: \"kubernetes.io/projected/cd085a4e-4d27-4397-b0db-7f010b6b4cad-kube-api-access-77v79\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:16.106511 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.106486    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"]
Oct 30 21:34:16.110965 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:16.110941    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-321ccd-5dd864d547-hg97g"]
Oct 30 21:34:17.661059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:17.661029    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="224bb67a-9e62-48c5-b703-a8de47cf7f92" path="/var/lib/kubelet/pods/224bb67a-9e62-48c5-b703-a8de47cf7f92/volumes"
Oct 30 21:34:17.661458 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:17.661416    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd085a4e-4d27-4397-b0db-7f010b6b4cad" path="/var/lib/kubelet/pods/cd085a4e-4d27-4397-b0db-7f010b6b4cad/volumes"
Oct 30 21:34:19.059084 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:19.059051    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"]
Oct 30 21:34:19.071978 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:19.071951    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9108683c-sdbdm"]
Oct 30 21:34:19.660926 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:19.660896    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e20dea9-a72f-4811-a049-a8fcab2540ec" path="/var/lib/kubelet/pods/8e20dea9-a72f-4811-a049-a8fcab2540ec/volumes"
Oct 30 21:34:20.141004 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.140956    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" containerID="cri-o://8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31" gracePeriod=45
Oct 30 21:34:20.145992 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.145588    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main" containerID="cri-o://ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d" gracePeriod=45
Oct 30 21:34:20.165972 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.165936    4810 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main" containerID="cri-o://cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74" gracePeriod=45
Oct 30 21:34:20.300147 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.300106    4810 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 21:34:20.431664 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.431342    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 21:34:20.463580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.463557    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 21:34:20.471091 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.471065    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 21:34:20.547489 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547455    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-home\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-home\") pod \"c71e8b54-b7bf-4df7-bd10-186443f455ee\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") "
Oct 30 21:34:20.547705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547497    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-dshm\") pod \"34915455-2f8c-46e5-ac31-89605f8ffe94\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") "
Oct 30 21:34:20.547705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547533    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71e8b54-b7bf-4df7-bd10-186443f455ee-kube-api-access-4lf7g\" (UniqueName: \"kubernetes.io/projected/c71e8b54-b7bf-4df7-bd10-186443f455ee-kube-api-access-4lf7g\") pod \"c71e8b54-b7bf-4df7-bd10-186443f455ee\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") "
Oct 30 21:34:20.547705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547578    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-dshm\") pod \"c71e8b54-b7bf-4df7-bd10-186443f455ee\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") "
Oct 30 21:34:20.547705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547603    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-tmp-dir\") pod \"34915455-2f8c-46e5-ac31-89605f8ffe94\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") "
Oct 30 21:34:20.547705 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547628    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34915455-2f8c-46e5-ac31-89605f8ffe94-tls-certs\" (UniqueName: \"kubernetes.io/secret/34915455-2f8c-46e5-ac31-89605f8ffe94-tls-certs\") pod \"34915455-2f8c-46e5-ac31-89605f8ffe94\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") "
Oct 30 21:34:20.548064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547717    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-home\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-home\") pod \"ab0e213f-d8cf-4659-829f-d7629ace07d4\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") "
Oct 30 21:34:20.548064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547769    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0e213f-d8cf-4659-829f-d7629ace07d4-kube-api-access-fk5nh\" (UniqueName: \"kubernetes.io/projected/ab0e213f-d8cf-4659-829f-d7629ace07d4-kube-api-access-fk5nh\") pod \"ab0e213f-d8cf-4659-829f-d7629ace07d4\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") "
Oct 30 21:34:20.548064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547800    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-tmp-dir\") pod \"c71e8b54-b7bf-4df7-bd10-186443f455ee\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") "
Oct 30 21:34:20.548064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547816    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-home" pod "c71e8b54-b7bf-4df7-bd10-186443f455ee" (UID: "c71e8b54-b7bf-4df7-bd10-186443f455ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.548064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547850    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-dshm\") pod \"ab0e213f-d8cf-4659-829f-d7629ace07d4\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") "
Oct 30 21:34:20.548064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547910    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-tmp-dir\") pod \"ab0e213f-d8cf-4659-829f-d7629ace07d4\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") "
Oct 30 21:34:20.548064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547945    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab0e213f-d8cf-4659-829f-d7629ace07d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0e213f-d8cf-4659-829f-d7629ace07d4-tls-certs\") pod \"ab0e213f-d8cf-4659-829f-d7629ace07d4\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") "
Oct 30 21:34:20.548064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.547994    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-model-cache\") pod \"34915455-2f8c-46e5-ac31-89605f8ffe94\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") "
Oct 30 21:34:20.548064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.548031    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34915455-2f8c-46e5-ac31-89605f8ffe94-kube-api-access-bkqcp\" (UniqueName: \"kubernetes.io/projected/34915455-2f8c-46e5-ac31-89605f8ffe94-kube-api-access-bkqcp\") pod \"34915455-2f8c-46e5-ac31-89605f8ffe94\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") "
Oct 30 21:34:20.548064 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.548056    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c71e8b54-b7bf-4df7-bd10-186443f455ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/c71e8b54-b7bf-4df7-bd10-186443f455ee-tls-certs\") pod \"c71e8b54-b7bf-4df7-bd10-186443f455ee\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") "
Oct 30 21:34:20.548622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.548106    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-home\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-home\") pod \"34915455-2f8c-46e5-ac31-89605f8ffe94\" (UID: \"34915455-2f8c-46e5-ac31-89605f8ffe94\") "
Oct 30 21:34:20.548622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.548139    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-model-cache\") pod \"ab0e213f-d8cf-4659-829f-d7629ace07d4\" (UID: \"ab0e213f-d8cf-4659-829f-d7629ace07d4\") "
Oct 30 21:34:20.548622 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.548179    4810 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-model-cache\") pod \"c71e8b54-b7bf-4df7-bd10-186443f455ee\" (UID: \"c71e8b54-b7bf-4df7-bd10-186443f455ee\") "
Oct 30 21:34:20.548791 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.548655    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.549473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.548099    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-home" pod "ab0e213f-d8cf-4659-829f-d7629ace07d4" (UID: "ab0e213f-d8cf-4659-829f-d7629ace07d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.549473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.548289    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-tmp-dir" pod "34915455-2f8c-46e5-ac31-89605f8ffe94" (UID: "34915455-2f8c-46e5-ac31-89605f8ffe94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.549473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.548907    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-model-cache" pod "c71e8b54-b7bf-4df7-bd10-186443f455ee" (UID: "c71e8b54-b7bf-4df7-bd10-186443f455ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.549473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.549322    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-model-cache" pod "34915455-2f8c-46e5-ac31-89605f8ffe94" (UID: "34915455-2f8c-46e5-ac31-89605f8ffe94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.549473 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.549330    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-tmp-dir" pod "c71e8b54-b7bf-4df7-bd10-186443f455ee" (UID: "c71e8b54-b7bf-4df7-bd10-186443f455ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.549844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.549546    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-home" pod "34915455-2f8c-46e5-ac31-89605f8ffe94" (UID: "34915455-2f8c-46e5-ac31-89605f8ffe94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.549844 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.549746    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-tmp-dir" pod "ab0e213f-d8cf-4659-829f-d7629ace07d4" (UID: "ab0e213f-d8cf-4659-829f-d7629ace07d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.550000 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.549947    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-model-cache" pod "ab0e213f-d8cf-4659-829f-d7629ace07d4" (UID: "ab0e213f-d8cf-4659-829f-d7629ace07d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.550462 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.550435    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-dshm" pod "34915455-2f8c-46e5-ac31-89605f8ffe94" (UID: "34915455-2f8c-46e5-ac31-89605f8ffe94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.550813 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.550788    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34915455-2f8c-46e5-ac31-89605f8ffe94-tls-certs" pod "34915455-2f8c-46e5-ac31-89605f8ffe94" (UID: "34915455-2f8c-46e5-ac31-89605f8ffe94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:20.551152 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.551097    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71e8b54-b7bf-4df7-bd10-186443f455ee-kube-api-access-4lf7g" pod "c71e8b54-b7bf-4df7-bd10-186443f455ee" (UID: "c71e8b54-b7bf-4df7-bd10-186443f455ee"). InnerVolumeSpecName "kube-api-access-4lf7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:20.551152 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.551139    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-dshm" pod "c71e8b54-b7bf-4df7-bd10-186443f455ee" (UID: "c71e8b54-b7bf-4df7-bd10-186443f455ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.551370 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.551347    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0e213f-d8cf-4659-829f-d7629ace07d4-kube-api-access-fk5nh" pod "ab0e213f-d8cf-4659-829f-d7629ace07d4" (UID: "ab0e213f-d8cf-4659-829f-d7629ace07d4"). InnerVolumeSpecName "kube-api-access-fk5nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:20.551534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.551506    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34915455-2f8c-46e5-ac31-89605f8ffe94-kube-api-access-bkqcp" pod "34915455-2f8c-46e5-ac31-89605f8ffe94" (UID: "34915455-2f8c-46e5-ac31-89605f8ffe94"). InnerVolumeSpecName "kube-api-access-bkqcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:20.552149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.552125    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-dshm" pod "ab0e213f-d8cf-4659-829f-d7629ace07d4" (UID: "ab0e213f-d8cf-4659-829f-d7629ace07d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:20.552574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.552547    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab0e213f-d8cf-4659-829f-d7629ace07d4-tls-certs" pod "ab0e213f-d8cf-4659-829f-d7629ace07d4" (UID: "ab0e213f-d8cf-4659-829f-d7629ace07d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:20.552574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.552559    4810 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71e8b54-b7bf-4df7-bd10-186443f455ee-tls-certs" pod "c71e8b54-b7bf-4df7-bd10-186443f455ee" (UID: "c71e8b54-b7bf-4df7-bd10-186443f455ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:20.650030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.649991    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0e213f-d8cf-4659-829f-d7629ace07d4-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650023    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650030 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650033    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkqcp\" (UniqueName: \"kubernetes.io/projected/34915455-2f8c-46e5-ac31-89605f8ffe94-kube-api-access-bkqcp\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650049    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71e8b54-b7bf-4df7-bd10-186443f455ee-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650059    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650067    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650076    4810 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-model-cache\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650085    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650094    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lf7g\" (UniqueName: \"kubernetes.io/projected/c71e8b54-b7bf-4df7-bd10-186443f455ee-kube-api-access-4lf7g\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650101    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650109    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34915455-2f8c-46e5-ac31-89605f8ffe94-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650119    4810 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34915455-2f8c-46e5-ac31-89605f8ffe94-tls-certs\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650130    4810 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-home\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650138    4810 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk5nh\" (UniqueName: \"kubernetes.io/projected/ab0e213f-d8cf-4659-829f-d7629ace07d4-kube-api-access-fk5nh\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650147    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71e8b54-b7bf-4df7-bd10-186443f455ee-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650155    4810 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-dshm\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:20.650295 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:20.650162    4810 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0e213f-d8cf-4659-829f-d7629ace07d4-tmp-dir\") on node \"ip-10-0-128-161.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:21.098541 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.098503    4810 generic.go:356] "Generic (PLEG): container finished" podID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerID="cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74" exitCode=0
Oct 30 21:34:21.098721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.098567    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"
Oct 30 21:34:21.098721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.098581    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" event={"ID":"ab0e213f-d8cf-4659-829f-d7629ace07d4","Type":"ContainerDied","Data":"cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74"}
Oct 30 21:34:21.098721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.098621    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb" event={"ID":"ab0e213f-d8cf-4659-829f-d7629ace07d4","Type":"ContainerDied","Data":"59b4e34064240aef7bdb2c4da2c57988147db31c0850b5fcf6ea7f6ee2b7bde8"}
Oct 30 21:34:21.098721 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.098641    4810 scope.go:122] "RemoveContainer" containerID="cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74"
Oct 30 21:34:21.100149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.100128    4810 generic.go:356] "Generic (PLEG): container finished" podID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerID="8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31" exitCode=0
Oct 30 21:34:21.100239 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.100185    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"
Oct 30 21:34:21.100312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.100229    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" event={"ID":"c71e8b54-b7bf-4df7-bd10-186443f455ee","Type":"ContainerDied","Data":"8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31"}
Oct 30 21:34:21.100312 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.100281    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx" event={"ID":"c71e8b54-b7bf-4df7-bd10-186443f455ee","Type":"ContainerDied","Data":"d7369f54d092be15930f61585c97f2a4e05a1cb6360c469661550d52426a573c"}
Oct 30 21:34:21.101732 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.101710    4810 generic.go:356] "Generic (PLEG): container finished" podID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerID="ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d" exitCode=0
Oct 30 21:34:21.101884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.101755    4810 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"
Oct 30 21:34:21.101884 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.101839    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" event={"ID":"34915455-2f8c-46e5-ac31-89605f8ffe94","Type":"ContainerDied","Data":"ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d"}
Oct 30 21:34:21.102020 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.101902    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9" event={"ID":"34915455-2f8c-46e5-ac31-89605f8ffe94","Type":"ContainerDied","Data":"33b53dfbff47c61998ac33abff7f611672cc89056e25c1c2a524152573ee5043"}
Oct 30 21:34:21.111602 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.111580    4810 scope.go:122] "RemoveContainer" containerID="cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74"
Oct 30 21:34:21.111912 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:34:21.111892    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74\": container with ID starting with cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74 not found: ID does not exist" containerID="cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74"
Oct 30 21:34:21.111989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.111921    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74"} err="failed to get container status \"cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74\": rpc error: code = NotFound desc = could not find container \"cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74\": container with ID starting with cf3340ab414c1326764064634e305c4c80db0f26448065bad075cca8562f8c74 not found: ID does not exist"
Oct 30 21:34:21.111989 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.111937    4810 scope.go:122] "RemoveContainer" containerID="8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31"
Oct 30 21:34:21.122695 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.122674    4810 scope.go:122] "RemoveContainer" containerID="8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31"
Oct 30 21:34:21.122987 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:34:21.122966    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31\": container with ID starting with 8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31 not found: ID does not exist" containerID="8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31"
Oct 30 21:34:21.123033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.122995    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31"} err="failed to get container status \"8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31\": rpc error: code = NotFound desc = could not find container \"8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31\": container with ID starting with 8aa3f6e930a44cf3b363ff9e7e13aa65e89574863b7324905840a91fdf04da31 not found: ID does not exist"
Oct 30 21:34:21.123033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.123011    4810 scope.go:122] "RemoveContainer" containerID="ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d"
Oct 30 21:34:21.130792 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.130765    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"]
Oct 30 21:34:21.132623 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.132605    4810 scope.go:122] "RemoveContainer" containerID="ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d"
Oct 30 21:34:21.132938 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:34:21.132920    4810 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d\": container with ID starting with ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d not found: ID does not exist" containerID="ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d"
Oct 30 21:34:21.133003 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.132944    4810 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d"} err="failed to get container status \"ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d\": rpc error: code = NotFound desc = could not find container \"ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d\": container with ID starting with ba5180167b5f15ad8380c6f8580ad46f3b31940e0f7a14ba804269bfdbe77a4d not found: ID does not exist"
Oct 30 21:34:21.138246 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.138225    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/simulator-w1-317a33-kserve-8c7446bf8-mdzwb"]
Oct 30 21:34:21.150936 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.150910    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"]
Oct 30 21:34:21.154354 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.154332    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-2-w1-317a33-kserve-7486c95665-6xjmx"]
Oct 30 21:34:21.165883 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.165845    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"]
Oct 30 21:34:21.170028 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.170008    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-317a33/distinct-w1-317a33-kserve-5f6f556978-msxc9"]
Oct 30 21:34:21.661695 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.661663    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" path="/var/lib/kubelet/pods/34915455-2f8c-46e5-ac31-89605f8ffe94/volumes"
Oct 30 21:34:21.662121 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.662107    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" path="/var/lib/kubelet/pods/ab0e213f-d8cf-4659-829f-d7629ace07d4/volumes"
Oct 30 21:34:21.662496 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:21.662483    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" path="/var/lib/kubelet/pods/c71e8b54-b7bf-4df7-bd10-186443f455ee/volumes"
Oct 30 21:34:23.577327 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:23.577275    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:34:25.157263 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:34:25.157226    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:34:28.986834 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:28.986800    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qwdhc"]
Oct 30 21:34:30.032010 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:30.031967    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"]
Oct 30 21:34:30.035251 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:30.035224    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0acf2aa-47tpf"]
Oct 30 21:34:31.660955 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:31.660918    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227cec8e-eed7-4036-84d4-7c3caabd5923" path="/var/lib/kubelet/pods/227cec8e-eed7-4036-84d4-7c3caabd5923/volumes"
Oct 30 21:34:35.345434 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:35.345403    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-cs5q5_b18bb0ae-131b-473b-b82e-e98ba3633842/kuadrant-console-plugin/0.log"
Oct 30 21:34:35.588753 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:35.588713    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7rkvg_c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8/manager/0.log"
Oct 30 21:34:35.822033 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:35.822003    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qwdhc_bcc30848-8e00-4f89-9f7b-a9f340c43866/limitador/0.log"
Oct 30 21:34:36.024133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:36.024064    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"]
Oct 30 21:34:36.027170 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:36.027143    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bca2ab3-kkkjk"]
Oct 30 21:34:37.381806 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:37.381771    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qnpf4_0fc88716-17fa-4b87-910f-a7ac94efc038/manager/0.log"
Oct 30 21:34:37.664873 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:37.664786    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62e800f-d9a7-4539-84f6-ff97f5229123" path="/var/lib/kubelet/pods/a62e800f-d9a7-4539-84f6-ff97f5229123/volumes"
Oct 30 21:34:49.022283 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:49.022248    4810 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"]
Oct 30 21:34:49.025069 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:49.025042    4810 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde3b9a6-gqghn"]
Oct 30 21:34:49.661369 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:49.661337    4810 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1115536-edb7-4c90-a90a-6015b87dff41" path="/var/lib/kubelet/pods/f1115536-edb7-4c90-a90a-6015b87dff41/volumes"
Oct 30 21:34:50.219626 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:50.219592    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qnpf4_0fc88716-17fa-4b87-910f-a7ac94efc038/manager/0.log"
Oct 30 21:34:52.016906 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:52.016870    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-cbf9f78f-j5g8h_9d56832d-9ea5-44de-a9cf-00d12d21f766/maas-api/0.log"
Oct 30 21:34:52.224514 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:52.224471    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-02aabb1d-rvxkb_33b09af8-6024-4169-82ea-98697ee8265f/revoke-keys/0.log"
Oct 30 21:34:52.572587 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:52.572561    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0d1a5c3d-7598d_e8954104-beb7-4d35-96bb-f71f0df8e873/revoke-keys/0.log"
Oct 30 21:34:53.133810 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:53.133776    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b01c180f-27pzz_3d6c4c8b-8e90-4f30-a445-35ee31746760/revoke-keys/0.log"
Oct 30 21:34:53.317149 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:53.317119    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b07a74d0-q2c4g_54769587-dc35-44e4-9f79-7701a79f4aa0/revoke-keys/0.log"
Oct 30 21:34:53.682574 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:53.682542    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bfd5ac09-nvzxg_2ba93cab-0db4-4f3f-869a-967924134e51/revoke-keys/0.log"
Oct 30 21:34:53.849803 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:53.849776    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cd30de49-k74nc_90687157-a544-4113-aae2-b4c0278794d4/revoke-keys/0.log"
Oct 30 21:34:54.030376 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:54.030299    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d41b309d-c7jkz_49f45b0d-c112-4592-9d24-1f1002eea388/revoke-keys/0.log"
Oct 30 21:34:55.156142 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:34:55.156054    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:34:56.444809 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:56.444778    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-cs5q5_b18bb0ae-131b-473b-b82e-e98ba3633842/kuadrant-console-plugin/0.log"
Oct 30 21:34:56.700027 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:56.699942    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7rkvg_c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8/manager/0.log"
Oct 30 21:34:56.956746 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:56.956667    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qwdhc_bcc30848-8e00-4f89-9f7b-a9f340c43866/limitador/0.log"
Oct 30 21:34:58.271577 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:58.271548    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-k2n4q_e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33/discovery/0.log"
Oct 30 21:34:58.623898 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:58.623806    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64cbf78866-vhxkm_06a067b9-1e21-4355-b407-5ed910022940/kube-auth-proxy/0.log"
Oct 30 21:34:58.981478 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:58.981452    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-d5dmn_f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30/istio-proxy/0.log"
Oct 30 21:34:59.170778 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:34:59.170754    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-f8lmw_9e300ef6-e571-4178-9b9b-b7a558122275/istio-proxy/0.log"
Oct 30 21:35:00.259166 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:00.259140    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dc99f8759-pzknq_6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/router/0.log"
Oct 30 21:35:17.638133 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:17.638098    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vmcq2_a248c2d8-ba41-4190-9ca7-6cce702dd07a/global-pull-secret-syncer/0.log"
Oct 30 21:35:17.685503 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:17.685481    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jr5cw_8a707e7f-24ce-4322-b5ad-ffaaa21dc23c/connection-checker/0.log"
Oct 30 21:35:17.729647 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:17.729621    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nbh76_a3f5256b-0e1f-4bf0-a83f-b18fcdf6b99a/connection-checker/0.log"
Oct 30 21:35:17.763987 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:17.763968    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-sp5fq_7fc47c20-4a62-4167-9334-4d7b1cadb5b5/connection-checker/0.log"
Oct 30 21:35:17.993059 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:17.992991    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vgcxk_51d19e04-9e59-4f4f-be5c-9900cca6d655/konnectivity-agent/0.log"
Oct 30 21:35:18.029250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:18.029226    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-161.us-east-2.compute.internal_157678277e297843d51e22f1021ace90/haproxy/0.log"
Oct 30 21:35:19.891480 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:19.891449    4810 scope.go:122] "RemoveContainer" containerID="c4f08b6ab49c1c0d39573aa992ce9fc417a796ac97cb9e2642f2716b6a4e7b50"
Oct 30 21:35:25.155298 ip-10-0-128-161 kubenswrapper[4810]: E1030 21:35:25.155262    4810 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kz7s8" containerName="extractor"
Oct 30 21:35:26.975474 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:26.975446    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-cs5q5_b18bb0ae-131b-473b-b82e-e98ba3633842/kuadrant-console-plugin/0.log"
Oct 30 21:35:27.076748 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:27.076718    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7rkvg_c317aeb3-a1fe-44b4-bb5a-243dcb95a2e8/manager/0.log"
Oct 30 21:35:27.544368 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:27.544337    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qwdhc_bcc30848-8e00-4f89-9f7b-a9f340c43866/limitador/0.log"
Oct 30 21:35:29.365274 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.365243    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-cd75b_47697b9d-ed21-4e3b-93b0-320138d4037d/cluster-monitoring-operator/0.log"
Oct 30 21:35:29.407241 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.407210    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h9kdb_c30e377f-0ccf-4b99-af6e-2711bf8f7bdb/kube-state-metrics/0.log"
Oct 30 21:35:29.438411 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.438379    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h9kdb_c30e377f-0ccf-4b99-af6e-2711bf8f7bdb/kube-rbac-proxy-main/0.log"
Oct 30 21:35:29.469737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.469713    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h9kdb_c30e377f-0ccf-4b99-af6e-2711bf8f7bdb/kube-rbac-proxy-self/0.log"
Oct 30 21:35:29.504728 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.504705    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f8657c7b9-5v8td_05f34d74-7115-42fb-a288-27bf8ef7c3e8/metrics-server/0.log"
Oct 30 21:35:29.537048 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.537027    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-84kx5_07c3a947-1296-4e31-9d93-4fdb854fc889/monitoring-plugin/0.log"
Oct 30 21:35:29.782358 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.782331    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tmxfc_e3e46551-28b8-47a0-ad33-59151c3137d2/node-exporter/0.log"
Oct 30 21:35:29.810100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.810072    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tmxfc_e3e46551-28b8-47a0-ad33-59151c3137d2/kube-rbac-proxy/0.log"
Oct 30 21:35:29.839013 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.838992    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tmxfc_e3e46551-28b8-47a0-ad33-59151c3137d2/init-textfile/0.log"
Oct 30 21:35:29.875056 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875032    4810 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"]
Oct 30 21:35:29.875523 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875509    4810 cpu_manager.go:414] "containerMap: removing container" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main"
Oct 30 21:35:29.875600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875524    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main"
Oct 30 21:35:29.875600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875539    4810 cpu_manager.go:414] "containerMap: removing container" podUID="1c20507b-2dd0-4490-b5eb-dc475f769a04" containerName="payload-processing"
Oct 30 21:35:29.875600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875544    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c20507b-2dd0-4490-b5eb-dc475f769a04" containerName="payload-processing"
Oct 30 21:35:29.875600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875554    4810 cpu_manager.go:414] "containerMap: removing container" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main"
Oct 30 21:35:29.875600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875560    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main"
Oct 30 21:35:29.875600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875578    4810 cpu_manager.go:414] "containerMap: removing container" podUID="4c34c2c8-319b-4173-9e40-f9abce02dee1" containerName="revoke-keys"
Oct 30 21:35:29.875600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875586    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c34c2c8-319b-4173-9e40-f9abce02dee1" containerName="revoke-keys"
Oct 30 21:35:29.875600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875596    4810 cpu_manager.go:414] "containerMap: removing container" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerName="istio-proxy"
Oct 30 21:35:29.875600 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875601    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerName="istio-proxy"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875611    4810 cpu_manager.go:414] "containerMap: removing container" podUID="0e74b3bb-da56-4fe7-9585-916338363efb" containerName="payload-pre-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875616    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e74b3bb-da56-4fe7-9585-916338363efb" containerName="payload-pre-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875626    4810 cpu_manager.go:414] "containerMap: removing container" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875631    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875638    4810 cpu_manager.go:414] "containerMap: removing container" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875642    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875655    4810 cpu_manager.go:414] "containerMap: removing container" podUID="297f5eb4-df4e-47dc-bb9b-41505d5da2be" containerName="payload-pre-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875663    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297f5eb4-df4e-47dc-bb9b-41505d5da2be" containerName="payload-pre-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875676    4810 cpu_manager.go:414] "containerMap: removing container" podUID="cd085a4e-4d27-4397-b0db-7f010b6b4cad" containerName="payload-pre-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875681    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd085a4e-4d27-4397-b0db-7f010b6b4cad" containerName="payload-pre-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875691    4810 cpu_manager.go:414] "containerMap: removing container" podUID="212bd647-b861-4804-88a0-a3cd4372ece2" containerName="payload-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875695    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212bd647-b861-4804-88a0-a3cd4372ece2" containerName="payload-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875702    4810 cpu_manager.go:414] "containerMap: removing container" podUID="224bb67a-9e62-48c5-b703-a8de47cf7f92" containerName="payload-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875706    4810 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224bb67a-9e62-48c5-b703-a8de47cf7f92" containerName="payload-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875787    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c20507b-2dd0-4490-b5eb-dc475f769a04" containerName="payload-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875795    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0e213f-d8cf-4659-829f-d7629ace07d4" containerName="main"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875802    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c34c2c8-319b-4173-9e40-f9abce02dee1" containerName="revoke-keys"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875810    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4234e57-d237-4b81-b8b6-a1ea7e75e12a" containerName="istio-proxy"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875821    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e74b3bb-da56-4fe7-9585-916338363efb" containerName="payload-pre-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875832    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71e8b54-b7bf-4df7-bd10-186443f455ee" containerName="main"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875838    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf96d91-a658-426b-8ee0-e471f8ef6730" containerName="istio-proxy"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875843    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="297f5eb4-df4e-47dc-bb9b-41505d5da2be" containerName="payload-pre-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875853    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd085a4e-4d27-4397-b0db-7f010b6b4cad" containerName="payload-pre-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875880    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="212bd647-b861-4804-88a0-a3cd4372ece2" containerName="payload-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875890    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="224bb67a-9e62-48c5-b703-a8de47cf7f92" containerName="payload-processing"
Oct 30 21:35:29.876035 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.875899    4810 memory_manager.go:365] "RemoveStaleState removing state" podUID="34915455-2f8c-46e5-ac31-89605f8ffe94" containerName="main"
Oct 30 21:35:29.881055 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.881037    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:29.889224 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.889199    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"]
Oct 30 21:35:29.916681 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.916658    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qtvts_a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54/kube-rbac-proxy-main/0.log"
Oct 30 21:35:29.950450 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.950405    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qtvts_a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54/kube-rbac-proxy-self/0.log"
Oct 30 21:35:29.975794 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.975764    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8ln49\"/\"openshift-service-ca.crt\""
Oct 30 21:35:29.975950 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.975891    4810 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8ln49\"/\"kube-root-ca.crt\""
Oct 30 21:35:29.976002 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.975950    4810 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8ln49\"/\"default-dockercfg-n87st\""
Oct 30 21:35:29.979244 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:29.979224    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qtvts_a52fc8a5-ace2-44a9-a2b1-7d4a9d2d4d54/openshift-state-metrics/0.log"
Oct 30 21:35:30.019656 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.019632    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-proc\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.019754 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.019688    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzhnx\" (UniqueName: \"kubernetes.io/projected/f5396dac-7d23-42b0-b537-85681755396e-kube-api-access-vzhnx\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.019798 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.019776    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-sys\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.019835 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.019812    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-podres\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.019922 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.019845    4810 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-lib-modules\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.120971 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.120913    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-proc\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.121076 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.120977    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzhnx\" (UniqueName: \"kubernetes.io/projected/f5396dac-7d23-42b0-b537-85681755396e-kube-api-access-vzhnx\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.121076 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.121011    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-sys\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.121076 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.121023    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-proc\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.121252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.121096    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-podres\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.121252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.121137    4810 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-lib-modules\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.121252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.121097    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-sys\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.121252 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.121198    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-podres\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.121435 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.121252    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5396dac-7d23-42b0-b537-85681755396e-lib-modules\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.129250 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.129219    4810 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzhnx\" (UniqueName: \"kubernetes.io/projected/f5396dac-7d23-42b0-b537-85681755396e-kube-api-access-vzhnx\") pod \"perf-node-gather-daemonset-whbjg\" (UID: \"f5396dac-7d23-42b0-b537-85681755396e\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.222286 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.222259    4810 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:30.232175 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.232133    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2fxhd_11c30c97-5275-4fc9-ab81-54c0f3de3ec2/prometheus-operator/0.log"
Oct 30 21:35:30.256102 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.256081    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2fxhd_11c30c97-5275-4fc9-ab81-54c0f3de3ec2/kube-rbac-proxy/0.log"
Oct 30 21:35:30.296100 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.296033    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-ssrdz_60a8f033-c381-47bd-aa2a-bf531bc18359/prometheus-operator-admission-webhook/0.log"
Oct 30 21:35:30.325156 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.325129    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fd6bd5bbd-j58w2_aa947095-7d5c-4e06-9e46-d297fe162fd3/telemeter-client/0.log"
Oct 30 21:35:30.352580 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.352551    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fd6bd5bbd-j58w2_aa947095-7d5c-4e06-9e46-d297fe162fd3/reload/0.log"
Oct 30 21:35:30.383230 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.383194    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fd6bd5bbd-j58w2_aa947095-7d5c-4e06-9e46-d297fe162fd3/kube-rbac-proxy/0.log"
Oct 30 21:35:30.429951 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.429930    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744b9bc9d7-grkth_af70ce71-5273-4ef0-908e-c374184d46a2/thanos-query/0.log"
Oct 30 21:35:30.458638 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.458618    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744b9bc9d7-grkth_af70ce71-5273-4ef0-908e-c374184d46a2/kube-rbac-proxy-web/0.log"
Oct 30 21:35:30.489290 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.489268    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744b9bc9d7-grkth_af70ce71-5273-4ef0-908e-c374184d46a2/kube-rbac-proxy/0.log"
Oct 30 21:35:30.496032 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.496012    4810 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"]
Oct 30 21:35:30.498900 ip-10-0-128-161 kubenswrapper[4810]: W1030 21:35:30.498872    4810 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf5396dac_7d23_42b0_b537_85681755396e.slice/crio-45da2f600252a88635cecb18cf8e155c7549458b53822da6086a0737dfd0a1bc WatchSource:0}: Error finding container 45da2f600252a88635cecb18cf8e155c7549458b53822da6086a0737dfd0a1bc: Status 404 returned error can't find the container with id 45da2f600252a88635cecb18cf8e155c7549458b53822da6086a0737dfd0a1bc
Oct 30 21:35:30.516891 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.516846    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744b9bc9d7-grkth_af70ce71-5273-4ef0-908e-c374184d46a2/prom-label-proxy/0.log"
Oct 30 21:35:30.546530 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.546505    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744b9bc9d7-grkth_af70ce71-5273-4ef0-908e-c374184d46a2/kube-rbac-proxy-rules/0.log"
Oct 30 21:35:30.550998 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.550972    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg" event={"ID":"f5396dac-7d23-42b0-b537-85681755396e","Type":"ContainerStarted","Data":"45da2f600252a88635cecb18cf8e155c7549458b53822da6086a0737dfd0a1bc"}
Oct 30 21:35:30.572271 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:30.572249    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744b9bc9d7-grkth_af70ce71-5273-4ef0-908e-c374184d46a2/kube-rbac-proxy-metrics/0.log"
Oct 30 21:35:31.559578 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:31.559545    4810 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg" event={"ID":"f5396dac-7d23-42b0-b537-85681755396e","Type":"ContainerStarted","Data":"04b3a4ecd7ba2613657460965da9a2b446221b057a725df86671360fc4b2ee73"}
Oct 30 21:35:31.560002 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:31.559635    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:31.577145 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:31.577100    4810 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg" podStartSLOduration=2.577087497 podStartE2EDuration="2.577087497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:31.574967205 +0000 UTC m=+3498.066783751" watchObservedRunningTime="2026-10-30 21:35:31.577087497 +0000 UTC m=+3498.068904066"
Oct 30 21:35:31.983849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:31.983817    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-wt7nr_734323fa-6946-426b-9e06-396cb4e7b48f/networking-console-plugin/0.log"
Oct 30 21:35:32.814661 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:32.814635    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/2.log"
Oct 30 21:35:32.818220 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:32.818192    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8nltv_ebc0553b-387f-438a-ab73-a053b37abcdd/console-operator/3.log"
Oct 30 21:35:33.630292 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:33.630255    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-g57td_d9f77f82-84fd-496c-994e-7fdc2a018af3/download-server/0.log"
Oct 30 21:35:34.610051 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:34.610022    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-klqns_0d7898c1-9d6c-4382-baa2-e6baae34941a/volume-data-source-validator/0.log"
Oct 30 21:35:34.947107 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:34.947070    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qnpf4_0fc88716-17fa-4b87-910f-a7ac94efc038/manager/0.log"
Oct 30 21:35:34.972316 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:34.972282    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qnpf4_0fc88716-17fa-4b87-910f-a7ac94efc038/manager/0.log"
Oct 30 21:35:36.666839 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:36.666808    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nfvzn_6140cde0-b52b-48e1-a1d6-bcac35086922/dns/0.log"
Oct 30 21:35:36.695198 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:36.695174    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nfvzn_6140cde0-b52b-48e1-a1d6-bcac35086922/kube-rbac-proxy/0.log"
Oct 30 21:35:36.835405 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:36.835380    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4mv46_7389168a-7f19-48df-98f8-6d6d20892416/dns-node-resolver/0.log"
Oct 30 21:35:37.576821 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:37.576793    4810 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-whbjg"
Oct 30 21:35:37.875534 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:37.875452    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56dfd5c8cd-9dzxh_baa5eb14-051a-4be5-be3a-bc23b427540e/registry/0.log"
Oct 30 21:35:37.976191 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:37.976158    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sq5lx_58b187ea-dead-4a37-8a52-5e447783d815/node-ca/0.log"
Oct 30 21:35:39.981375 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:39.981336    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qnpf4_0fc88716-17fa-4b87-910f-a7ac94efc038/manager/0.log"
Oct 30 21:35:40.078913 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:40.078848    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qnpf4_0fc88716-17fa-4b87-910f-a7ac94efc038/manager/0.log"
Oct 30 21:35:42.854840 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:42.854807    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-k2n4q_e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33/discovery/0.log"
Oct 30 21:35:43.322995 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:43.322967    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64cbf78866-vhxkm_06a067b9-1e21-4355-b407-5ed910022940/kube-auth-proxy/0.log"
Oct 30 21:35:43.411814 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:43.411783    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-d5dmn_f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30/istio-proxy/0.log"
Oct 30 21:35:43.447899 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:43.447849    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-f8lmw_9e300ef6-e571-4178-9b9b-b7a558122275/istio-proxy/0.log"
Oct 30 21:35:43.696243 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:43.696218    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dc99f8759-pzknq_6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/router/0.log"
Oct 30 21:35:45.453849 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:45.453795    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qnpf4_0fc88716-17fa-4b87-910f-a7ac94efc038/manager/0.log"
Oct 30 21:35:45.586001 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:45.585967    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qnpf4_0fc88716-17fa-4b87-910f-a7ac94efc038/manager/0.log"
Oct 30 21:35:46.617238 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:46.617207    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-k2n4q_e21e1a6b-45b8-48ef-90b6-ff19dd0c7e33/discovery/0.log"
Oct 30 21:35:46.664542 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:46.664519    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64cbf78866-vhxkm_06a067b9-1e21-4355-b407-5ed910022940/kube-auth-proxy/0.log"
Oct 30 21:35:46.727567 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:46.727540    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-d5dmn_f0ffe7b5-9ec3-4a68-acfc-df7dc69b5c30/istio-proxy/0.log"
Oct 30 21:35:46.748944 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:46.748920    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-f8lmw_9e300ef6-e571-4178-9b9b-b7a558122275/istio-proxy/0.log"
Oct 30 21:35:46.918990 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:46.918962    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dc99f8759-pzknq_6b0e6adb-30a6-481c-8ee0-27c52c7c2fe2/router/0.log"
Oct 30 21:35:46.943390 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:46.943365    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9v7g_93f061fe-5c27-419b-8677-9734d5dce379/serve-healthcheck-canary/0.log"
Oct 30 21:35:47.871400 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:47.871367    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-rbf5z_b3e2fc05-6adb-4c94-a124-aca4483a4e13/insights-operator/1.log"
Oct 30 21:35:47.872472 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:47.872450    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-rbf5z_b3e2fc05-6adb-4c94-a124-aca4483a4e13/insights-operator/2.log"
Oct 30 21:35:48.012420 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:48.012397    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kz7s8_585e93ee-0fc2-4461-b5ad-e9e0f50afc06/kube-rbac-proxy/0.log"
Oct 30 21:35:48.041642 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:48.041621    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kz7s8_585e93ee-0fc2-4461-b5ad-e9e0f50afc06/exporter/0.log"
Oct 30 21:35:48.076737 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:48.076717    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kz7s8_585e93ee-0fc2-4461-b5ad-e9e0f50afc06/extractor/0.log"
Oct 30 21:35:48.221339 ip-10-0-128-161 kubenswrapper[4810]: I1030 21:35:48.221315    4810 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-kl7gz-rqftw_23526b46-033a-49da-bbaa-72977ecc7645/insights-gathering/0.log"